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Applications  of  educational  technology  thus  far  have 
not  successfully  harnessed  the  power  of  our  knowledge  of  be- 
havior.   Attempts  to  individualize  instruction  through  teach- 
ing  machines,  computer-assisted  instruction  and  programed 
learning  have  produced  equivocal  results  at  prohibitive  ex- 
pense.   Keller's  introduction  of  a  personalized  approach  to 
college  undergraduate  instruction  opened  new  possibilities 
for  practical  application  of  educational  technology.    In  his 
model,  the  extensive  monitoring  and  feedback  functions  of 
individualized  instruction  are  carried  out  by  student  proc- 
tors who  have  demonstrated  mastery  of  the  subject  matter  in 
a  previous  course,     The  present  study  evaluates  a  modifica- 
tion of  Keller's  proctoring  system,  with  a  view  to  providing 
an  alternative  that  will  be,  for  some  users,  more  feasible 
to  administer, 
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Method 

The  performance  of  two  groups  is  compared  on  20  per- 
formance samples  of  the  kind  developed  by  Johnston  and 
Pennypacker  (direct  continuous  recording  of  oral  responses 
to  stimulus  items  taken  from  text  and  lectures,  yielding 
rates  of  correct  and  incorrect  responding).     The  Control 
group  uses  previously  trained  proctors  of  the  type  employed 
by  Keller.     The  Classroom  group  uses  currently  enrolled 
students;  i.e.,  the  students  enrolled  in  the  course  proctor 
each  other.    To  control  for  the  benefits  of  practice  that 
one  student  might  derive  from  auditing  the  performance  of 
another,  alternate  test  forms  are  used.    Additionally,  a 
performance  validation  procedure  is  designed  into  the  ex- 
periment to  control  for  possible  collaboration  between  stu- 
dents. 
Results 

1.  The  Classroom  group  achieved  reliably  superior 
performance  rates, 

2.  The  effects  of  practice  on  specific  stimulus  items 
and  the  effects  of  collaboration  were  ruled  out  as  possible 
sources  of  the  difference  between  groucs. 

Cpnclus ions 

1.  The  self-contained  classroom  model  is  an  acceptable 
pragmatic  alternative  to  Keller's  model. 

2.  Students  who  act  an  interested  listeners  (proctors) 
and  responders  achieve  better  performance  rates  than  those 
who  act  as  responders  alone. 


CHAPTER  I 


INTRODUCTION 

The  need  for  improved  instructional  technology  appears 
to  be  continuous,  renewable  from  one  generation  to  the  next 
as  a  function  of  the  increasing  complexity  of  society. 
While  American  education  has  been  generally  responsive  to 
the  technological  revolution  of  the  past  two  decades--for 
which  Skinner's  well  known  contribution  (195*0  has  provided 
the  main  impetus — early  prophesies  of  sweeping  changes  in 
classrocm  technique  have  not  been  fulfilled.     The  task  of 
translating  theory  into  practice  has  turned  out  to  be  more 
difficult  than  expected.    Sckert  and  Neale's  review  of  the 
literature  (1965)  has  found  the  contribution  of  the  new 
technology  to  be  "quite  modest"  in  various  applied  situa- 
tions; Pressey  (196^)  has  referred  to  some  programs  as  edu- 
cational monstrosities;  Oettinger  and  Marks  (1968)  have  ex- 
pressed doubt  that  educational  technology  will  become  estab- 
lished in  the  schools  during  the  twentieth  century. 

The  need  for  an  effective  technology  nonetheless  re- 
mains.   Koraoski  (1969)  has  summarized  the  present  state  of 
affairs  by  urging  that  three  admissions  be  made:  First, 
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that  traditional  methods  are  inadequate  to  handle  the  indiv- 
idual needs  of  today's  learners;  second,  that  the  new  tech- 
nology is  inadequate  also,  hut  holds  better  promise  for  the 
future;  and  third,  that  the  best  way  to  bring  that  promise 
to  fruition  is  to  use  the  new  technology  as  broadly  as  poss- 
ible.    In  the  present  study  the  new  technology  is  used  in  a 
college  classroom  setting.    Well  established  principles  from 
educational  technology  are  combined  with  a  performance  mea- 
sure that  hitherto  has  not  been  widely  used  in  college  class- 
rooms.   This  is  the  direct  continuous  recording  of  the  verb- 
al responses  students  emit  when  presented  with  stimulus 
items  from  assigned  study  material.     The  reader  may  recog- 
nize the  recording  method  as  one  of  the  cornerstones  of  the 
experimental  analysis  of  behavior. 

£xpez:imental  behavior  analysis  has  played  a  prominent 
role  in  educational  technology  from  the  start.     Hanson  and 
Komoski  (1965),  writing  in  NEA's  Tg^hin^J.Iach.5.nes  and  Pro- 
grained  Learning.  II.  point  out  that  programed  instruction 
came  about  "because  of  the  willingness  of  a  number  of  exper- 
imental psychologists  to  extrapolate  from  research  with  low- 
er organisms  to  the  human  learning  situation"  (p.  649). 
Skinner  is  credited  with  being  the  first  to  employ  program- 
ing on  a  regular  basis  (at  Harvard  in  1957),  and  his  intro- 
duction of  the  teaching  machine  in  I954  is  generally  held 
to  be  of  greater  significance  than  Pressey's  work  with  class- 
room machines  some  JO  years  earlier.    This  is  not  due  to  any 
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intrinsic  merit  of  the  device  but  to  the  technology  which 
supports  it.     Skinner  himself  (1965)  makes  this  point  in 
stating  that  "teaching  machines  are  simply  devices  which 
make  it  possible  to  apply  technical  knowledge  of  human  be- 
havior to  the  practical  field  of  education"  (p,  168). 
Hence,  it  is  the  knowledge  of  human  behavior  that  is  criti- 
cal to  a  sound  educational  technology,  not  the  particular 
form  of  its  application.    What  is  needed  now  is  an  applica- 
tion that  meets  the  test  of  practicality,  for  it  is  just 
here  that  programed  learning,  teaching  machines  and  computer 
assisted  instruction  have  failed.     It  is  in  the  direction 
of  practicality  that  the  present  study  makes  its  main 
thrust. 

Statement  of  Problem  and  Purpose 

The  foregoing  paragraphs  may  be  summarized  and  rephrased 
to  form  a  preliminary  statement  of  the  problem  and  purpose 
of  this  study:  The  problem  is  that  applications  of  educa- 
tional technology  thus  far  have  not  successfully  harnessed 
the  power  of  our  knowledge  of  behavior;  the  purpose  of  the 
study  is  to  investigate  an  application  of  the  procedures  of 
behavior  analysis  in  undergraduate  university  instruction. 

Review  of ..^j^a^e^g^gp^^h 
The  reference  study  for  this  research  is  an  article  by 
Keller  (1968a)  in  which  the  groundwork  is  laid  for  a  new 


college  teaching  method  that  has  been  called  personalised, 
programed .  and  proctorial  (Keller,  1968b,  p.  1).    It  is  con- 
venient to  break  the  review  down  into  subtopics.     In  the 
first  section,  the  relationship  between  Keller's  method  and 
prior  educational  technology  is  established  through  discussion 
of  important  similarities  and  differences.    In  the  second 
section,  other  studies  dealing  with  contingency  management 
in  the  college  classroom  are  reviewed.    In  the  third,  a  ma- 
jor innovation  to  the  basic  system  is  discussed.     Prom  this 
the  problem  and  purpose  of  the  study  are  reformulated  in 
more  c  oner e  te  t e  r ms . 

Relation  to  Previous  Work 

In  describing  his  contingency  management  approach  to 
college  ins  true  tier.,  Keller  (1968a;  1968b)  acknowledges  the 
similarities  to  programed  instruction.    He  refers  to  "the 
same  stress  upon  analysis  of  the  task,  the  same  concern  with 
terminal  performance,  the  same  opportunity  for  individual 
progression"  (1968a,  pp.  83-84).     Other  features  of  his  sys- 
tem are  almost  identical  to  those  listed  in  standard  out- 
lines of  programed  learning  (cf.  Cook  and  Mechner,  1962): 
The  active  or  constructed  response  by  the  student,  immedi- 
ate feedback  on  responses,  successive  mastery  of  relatively 
small  units  of  material. 

The  units,  however,  differ  from  the  typical  frames  of 
programed  instruction  in  that  they  more  closely  resemble 
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conventional  homework  assignments  or  laboratory  exercises. 

The  "response"  is  not  simply  the  completion  of 
a  prepared  statement  through  the  insertion  of  a 
word  or  phrase.    Rather,  it  may  be  thought  of  as 
the  resultant  of  many  such  responses,  better  de- 
scribed as  the  understanding  of  a  principle,  a 
formula,  or  a  concept,  or  the  ability  to' use  an 
experimental  technique. (Keller,  1968a,  p.  84) 

The  position  revealed  here  follows  the  spirit  of  Pressey' s 
arguments  (1963?  1964a;  1964b;  196?)  for  variable  frame 
length — "from  a  single  statement  to  an  entire  chapter"  (1967, 
p.  239) — and  for  the  preservation  rather  than  replacement  of 
established  materials  of  instruction.    Pressey  sees  "adjunct 
auto-instructional  devices"  as  eminently  more  practical  than 
programing  of  the  type  exemplified  by  Holland  and  Skinner 
(I958)i  and  usually  more  effective.    The  question  of  rela- 
tive effectiveness  has  not  been  satisfactorily  resolved  in 
the  research  literature  (Holland,  1965;  Silberman,  I962), 
but  there  can  be  little  doubt  that  adjunct  auto-instruction 
is  more  practical.     Basically  it  is  a  testing  procedure.  It 
follows  the  presentation  of  the  material  to  be  learned  (lec- 
ture or  reading  assignments);  in  programed  instruction,  the 
program  is  the  material  to  be  learned.     The  purpose  of  the 
adjunct  method  is  to  "clarify  and  extend  the  meaningf ulness" 
of  the  basic  material  through  emphasis  of  key  points  and  re- 
structuring of  ideas  (Pressey,  1963,  p.  5).    As  a  practical 
matter,  the  importance  of  adjunct  auto-instruction  is  that 
it  places  no  constraint  upon  its  user  to  solve  the  mysteries 
of  program  structure,  sequencing  or  hierarchy  of  learning 
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tasks.     "Such  auto-elucidation,"  Pressey  says,  "will  not  cov 
er  everything,  may  jump  from  one  point  to  another  or  even 
back  and  forth  ..."  (1963,  p.  3). 

One  of  the  unsolved  problems  of  programed  instruction 
has  been  the  cost  in  time  and  money  of  developing  programs. 
It  is  especially  acute  at  the  college  level  because  of  the 
complexity  of  the  subject  matter  and  its  tendency  toward 
early  obsolescence  (Green,  196?) .    Pressey's  adjunct  auto- 
instruction  is  one  way  out  of  the  dilemma,  even  though  it 
does  violence  to  Skinner's  concept  of  programed  instruction. 
In  Keller's  used  of  adjunct  materials,  the  break  from  earl- 
ier theory  is  less  painful.     The  auto-instructional  materi- 
als are  not  programed  in  the  Skinnerian  sense,  but  the  over- 
all system  is  based  predominantly  on  the  principles  of  be- 
havior analysis  that  apply  to  programed  learning.  Indeed, 
Keller  refers  to  it  as  a  "programed  system." 

Another  departure  .that  Keller  makes  from  earlier  models 
is  the  use  of  students  in  proctoring  roles.    The  manpower 
requirement  is  enormous  in  a  system  that  breaks  the  course 
material  into  many  small  units  and  requires  every  student 
to  demonstrate  mastery  of  each  unit  in  turn.  Alternate 
forms  of  the  performance  test  are  taken  as  many  times  as 
necessary  until  criterion  (mastery)  is  reached.     The  price 
of  individualized  instruction  is  high,  whether  it  be  carried 
out  through  low  student  ratios,  programed  instruction,  or 
the  computer.    Keller's  master  stroke  was  to  effectively 
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utilize  a  resource  that  was  available  at  no  cost.     In  so 
doing,  he  not  only  individualized  instruction,  but  personal- 
ized it  as  well. 

Related  Contingency  Management  Studies 

Since  the  inception  of  Keller's  work  in  1962,  several 
reports  of  similar  projects  have  appeared.  Unfortunately, 
most  of  these  have  been  demonstration  rather  than  research 
projects.     The  main  focus  in  the  following  review  is  on  the 
various  adjunct  auto-ins tructional  systems  used. 

Ferster  (I968)  developed  an  interview  technique  in 
which  students  responded  orally  to  study  questions  on  the 
textbook,  Behavior  Principles  (Ferster  and  Perrott,  1968). 
Each  student  had  the  responsibility  of  acting  as  a  listener 
for  another.    Satisfaction  with  the  interview  was  decided 
jointly  by  both  the  speaker  and  the  listener,  with  unsatis- 
factory performances  being  repeated.    Several  short  written 
quizzes  and  a  two  hour  final  examination  were  administered 
for  purposes  of  certifying  the  validity  of  interviews.  The 
grade,  however,  was  determined  by  the  number  of  interviews 
successfully  completed.    Fifty-nine  interviews  earned  the 
grade  of  "A"  and  percentages  of  that  number  earned  "3"  and 
"C".     In  the  class  reported,  90  percent  of  the  students  re- 
ceived the  grade  of  "A". 

A  group  interview  technique  has  been  used  by  Postle- 
thwait  and  Novak  (1967)  in  a  freshman  botany  course.    A  doz- 
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en  or  so  students  met  regularly  with  the  instructor  and  re- 
sponded orally  to  items  from  previous  study  assignments. 
Points  for  performance  were  awarded  by  the  instructor  on  a 
scale  from  zero  to  ten, 

i 

Kalott  and  Svinicki  (1969)  have  organized  an  introduc- 
tory psychology  course  along  contingency  management  lines. 
Quizzes  are  given  daily  to  cover  assigned  reading.  Each 
quiz  requires  written  answers  to  two  questions.     If  both  are 
not  answered  correctly,  the  student  attends  a  review  session 
and  takes  one  or  more  additional  quizzes  to  achieve  100  per- 
cent mastery.    Another  adjunctive  device  is  the  four-man 
discussion  group.    As  in  Ferster's  system,  the  purpose  is  to 
help  students  develop  oral  fluency  in  discussing  the  subject 
matter.    Peers  rate  each  other's  performance  in  the  discus- 
sion group,  and  instructional  personnel  monitor  interviews 
on  a  sampling  basis  only.    Teaching  apprentices,  equivalent 
to  Keller's  proctors,  are  drawn  from  previous  classes  and  re- 
ceive academic  credit  for  their  .services.    Fifty-two  of  them 
serve  the  typical  enrollment  of  1,000  students.  Addition- 
ally, 13  advanced  teacher  apprentices,  four  paid  assistants, 
and  a  full  time  secretary  assist  the  three  faculty  members 
who  conduct  the  course, 

Lloyd  and  Knutzen's  (1969)  course  in  the  experimental 
analysis  of  behavior  features  a  point  system,  or  token  econ- 
omy approach  to  contingency  management.    Thirty-five  students 
were  given  a  list  of  course  activities  at  the  beginning  of 
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the  term,  with  point  values  specified  for  each  activity. 
Grades  were  made  contingent  upon  the  activities  undertaken 
and  the  number  of  points  earned.    Several  activities  requir- 
ing constructed  responses  were  given  numeric  value  ratings 
in  the  manner  used  by  Postlethwait  and  Novak  (196?). 

One  of  the  few  studies  that  makes  an  experimental  com- 
parison between  a  contingency-managed  classroom  and  a  class 
taught  by  conventional  methods  has  been  reported  by  McMichael 
and  Corey  (i960).     They  found  that  learning,  as  reflected  in 
final  examination  scores,  was  significantly  better  in  a 
class  using  Keller's  methods  than  in  conventionally  taught 
classes.     The  finding  confirms  some  results  which  are  men- 
tioned by  Keller  (1968a),  but  which  are  not  published.  In 
the  experimental  section  of  IJcLiichael  and  Corey's  class, 
there  were  221  students.     Two  instructors  and  two  graduate 
students  were  assisted  by  19  undergraduate  proctors  in  ad- 
ministering the  experimental  program. 

of  Behavior  Analyticjjej^jo^np;^ 

The  use  of  rate  as  a  basic  datum  in  the  experimental 
analysis  of  behavior  is  well  known.     Its  use  in  describing 
behavior  of  interest  to  education  has  also  been  established 
(Lindsley,  196^).    Edinger  (I969)  has  recently  demonstrated 
the  utility  of  using  free  operant  techniques  in  the  measure- 
ment and  analysis  of  programed  material  used  by  elementary 
school  children.    However,  the  method  was  conspicuously  ab- 
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sent  from  contingency  managed  college  classrooms  until  Pen- 
nypacker and  his  associates  (1969)  developed  a  precision 
taught  course  in  experimental  behavior  analysis.  Regarding 
the  method  of  measurement,  Pennypacker  says: 

The  teaching  process  has  as  its  stated  objective 
the  generation  of  behavioral  change.     It  is  es- 
sential that  these  changes  in  behavior  be  mea- 
sured as  directly  and  continuously  as  possible. 
Direct  measurement  avoids  the  numerous  hazards 
of  psychometric  inference  while  continuous  meas- 
urement permits  constant  adjustment  of  curricu- 
lum as  necessary  to  obtain  the  stated  object- 
ives.    (Pennypacker  et  al,  1969,  p. 2) 

Pennypacker  uses  the  proctorial  system  to  meet  the  man- 
power requirements  of  direct  continuous  recording.     The  ad- 
junct materials  are  constructed  response  items  similar  to 
the  frames  of  programed  instruction.     They  are  typed  on 
3"  X  5"  flip  cards.    The  student,  in  the  presence  of  his  man- 
ager (proctor),  reads  the  statement  aloud,  supplies  the  miss- 
ing element,  then  flips  the  card  for  immediate  confirmation 
or  correction.    The  performance  is  timed.    Rates  correct  and 
incorrect  are  computed  by  dividing  the  number  of  responses 
(correct  and  incorrect,  respectively)  by  the  number  of  min- 
utes elapsed  (usually  about  five).    As  in  other  contingency 
management  models,  there  is  a  criterion  for  mastery,  and 
students  are  allowed  to  repeat  performances  in  order  to 
reach  criterion,     The  percentage  of  students  achieving  the 
grade  of  "A"  is  high,  about  90  percent,  as  in  the  courses  of 
Keller  (1963a),  Ferster  (1968),  and  Malott  and  Svinicki 
(1969). 


11 


In  a  more  recent  manuscript,  Johnston  and  Pennypacker 
(1970)  report  the  early  results  of  a  long  term  research  pro- 
gram  which  uses  the  method  referred  to  above.  Preliminary 
investigation  indicates  that  the  system  is  superior  to  con- 
ventional teaching  methods  in  the  quality  and  quantity  of 
learning  it  produces;  that  it  can  be  applied  to  subject  mat- 
ter outside  the  field  of  behavior  analysis;  that  it  v/orks  as 
well  with  written  response  modes  as  with  oral;  and  that  the 
recording  and  display  of  performance  measurement  may  be 
based  on  either  rate  or  percent,  depending  on  the  instruct- 
or's preference,  without  detriment  to  the  results.     Of  in- 
terest is  the  consistently  high  rating  the  course  is  given 
in  statements  of  student  preference.    However,  the  primary 
relevance  of  this  report  to  the  present  study  is  the  compre- 
hensive exposition  of  philosophy  and  method  that  it  contains. 
It  is  both  the  parent  system  and  point  of  departure  for  the 
present  study.    The  present  research  is  best  understood  in 
the  light  of  that  fact. 

Summary  of  the  Relevant  literature 

Several  statements  can  be  made  in  summarizing  the  fore- 
going j 

1.    The  operational  objective  of  educational  technology  is 
to  individualize  instruction.     Individualization  implies  con- 
tinuous monitoring— a  feedback  system  that  informs  the  in- 
structor as  well  as  the  learner  of  the  exact  point  in  the 
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curriculum  at  which  learning  (performance)  has  failed. 

2,  The  development  of  such  systems  has  been  slow,  partly 
because  of  the  equivocal  research  findings  on  variables  in 
human  learning  in  field  situations,  and  partly  because  of 
the  high  cost  of  application  (hardware  and  software). 

3.  Keller  and  his  followers  have  demonstrated  methods  of 
providing  a  continuous  feedback  system  at  minimal  cost, 
through  utilization  of  students  in  proctorial  roles. 

fr.    Concurrent  advances  in  experimental  behavior  analysis 
have  brought  a  powerful  research  tool  to  bear  on  problems  of 
human  learning. 

5.  In  Pennypacker's  system,  the  two  mainstreams — Keller's 
method  of  personalizing  instruction  with  student  proctors, 
and  the  direct  continuous  recording  procedures  of  behavior 
analysis— converge  to  produce  a  unique  behavioral  approach 
to  college  teaching. 

Restatement  of  the  Problem 
The  most  salient  characteristic  of  the  studies  on  con- 
tingency managed  college  classes  is  the  paucity  of  systemat- 
ic research.    Only  one  of  the  published  studies  evaluates 
the  method  experimentally.     This  is  a  gross  comparison  of 
Keller's  method  versus  conventional  methods.    The  effects  of 
components  within  the  system  have  not  been  adequately  inves- 
tigated.   In  the  present  study,  an  evaluation  of  one  of  the 
principal  components,  the  proc boring  method,  is  undertaken. 
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Some  of  the  assumptions  underlying  the  use  of  student  proc- 
tors are  tested.    To  isolate  the  experimental  variable,  it 
is  necessary  also  to  drop  other  components  from  the  usual 
format.    Foremost  among  these  is  the  unit  mastery  provision. 
Another  is  the  self-pacing  feature.    These  components  are 
discussed  in  more  detail  in  the  following  pages,  after  which 
the  research  problem  is  recast  in  a  series  of  experimental 
questions , 

The  Use  of  Previously  Trained  Proctors 

Student  proctors  are  referred  to  as  "knowledgeable 
peers"  in  the  Johnston  and  Pennypacker  (1970)  article.  In 
proctor  selection,  it  is  the  practice  to  accept  only  those 
who  have  demonstrated  their  expertise  with  the  subject  mat- 
ter by  earning  the  grade  of  "A".    Implicit  in  this  is  the 
assumption  that  the  currently  enrolled  student  gains  a  salu- 
tary benefit  from  the  experienced  proctor's  tutorial  skills. 
Keller  (1968a)  states  as  much,  and  others  appear  to  have  ac- 
cepted his  reasoning.     No  one  has  tested  the  assumption  em- 
pirically, however,  or  evaluated  alternatives  to  the  present 
proctoring  system.    There  are  several  reasons  for  doing  so: 

1.  The  system  wants  simplification  if  it  is  to  be 
widely  used.     Coordinating  the  acquisition  and  assignment  of 
proctors  is  the  kind  of  administrative  task  that  would  inhib- 
it some  potential  users. 

2.  There  is  some  question  whether  this  kind  of  activity 
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makes  the  best  use  of  the  student  proctor's  time.    It  is 
probably  true,  as  Keller  (1968a)  asserts,  that  a  proctoring 
assignment  cements  the  student's  command  of  the  material. 
This  may  be  desirable  in  a  course  on  the  principles  of  behav 
ior  analysis.     These  are  difficult  principles  to  grasp,  and 
overlearning  may  be  the  best  way  to  master  them.    But  this 
would  not  necessarily  hold  for  other  curricula.     The  curri- 
culum  used  in  the  present  study  is  a  case  in  point.     In  the 
introductory  course  for  Teaching  Exceptional  Children,  the 
units  are  more  discrete  than  sequential.    Mastery  of  the 
unit  on  the  blind  in  not  a  prerequisite  to  success  on  units 
for  the  deaf  or  mentally  retarded.     The . presentation  is  more 
horizontal  than  vertical,  and  there  is  little  reason  to  be- 
lieve that  the  proctor's  understanding  of  it  would  improve 
with  repetition.    He  might  be  better  served  by  pursuing  an 
interest  in  one  of  the  exceptionalities  in  depth. 

3.     It  is  difficult  to  achieve  uniformity  of  treatment 
from  one  proctor  to  the  next.    Many  test  items,  including 
some  of  the  most  carefully  worded,  will  evoke  responses  that 
are  neither  clearly  correct  nor  clearly  incorrect.    A  re- 
sponse judged  incorrect  by  one  proctor  may  be  accepted  by 
another.     In  a  system  that  has  the  student  reading  some  ^50 
to  500  items  per  quarter,  the  disposition  of  the  proctor  can 
be  critically  important.     The  problem  can  be  circumvented  to 
a  degree  by  conducting  the  performance  sessions  en  masse,  in 
the  manner  Ferster  (I96S)  used,  with  the  instructor  present 
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to  arbitrate  questionable  items.  With  one  person  making  the 
decisions  instead  of  five  or  six,  the  chances  for  uniformity 
improve , 

Restricting  Unit  Mastery    (Time-Based  versus  Movement-Based 
Systems 

The  basic  distinctions  between  traditional  evaluation 
systems  and  those  employed  in  the  new  educational  technology 
are  well  known.     One  seeks  to  reflect  individual  differences 
in  ability  by  producing  a  distribution  of  scores,  the  other 
seeks  to  obviate  those  difference  by  individualizing  the 
curriculum.     One  makes  judgments  about  the  quality  of  the 
student,  the  other  makes  judgments  about  the  quality  of  the 
instructional  program. 

Contingency  managed  college  classrooms  typically  achieve 
individualization  through  utilizing  movement-based  perform- 
ance modes.     In  the  normal  classroom  the  performance  mode  is 
essentially  time-based.    Students  take  lessons,  quizzes, 
exams  as  a  group.     The  time  for  doing  so  is  fixed  by  the  in- 
structor, and  the  measurement  of  interest  is  the  quality  of 
the  movement  within  the  set  time  frame.     Usually  this  is  ex- 
pressed as  number  or  percentage  correct.    Movement-based  sys- 
tems,   on  the  other  hand,  make  performance  (unit  mastery)  the 
inflexible  element.     The  time  it  takes  to  achieve  mastery  is 
the  manipulandum  that  neutralizes  the  effect  of  individual 
differences  in  ability  level.     Slower  students  simply  keep 
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working  on  a  unit  until  they  have  it  mastered.    Hence,  time- 
to-completion  becomes  distributed  while  quality  of  perform- 
ance is  held  constant.    The  former  case  is  said  to  be  in- 
structor-paced, the  latter  student-  or  self-paced. 

Administratively,  the  movement-based  system  presents 
some  problems.    A  distribution  of  times-to-completion  im- 
plies multiple  exit.    Some  students  finish  in  less  than  the 
usual  time,  others  have  to  take  the  grade  of  "I"  (Incomplete) 
and  finish  the  work  in  a  succeeding  term.    Keller  (1968a) 
mentions  a  student  who  took  nearly  two  terms  to  complete  the 
work  of  one  and  then  became  a  proctor.    Malott  and  Svinicki 
(1969),  unlike  Keller,  attempt  to  control  the  distribution 
of  times-to-completion  by  governing  the  rate  at  which  quiz- 
zes are  given  on  the  reading  assignments.    Performance  is 
not  completely  ins truct or- paced ,  however: 

.  .  .  the  rate  a  student  covers  a  specific  assign- 
ment is  determined  by  the  student.    In  this  sense 
it  is  "student  paced."    In  other  words  cne  student 
may  need  to  spend  only  15  minutes  on  the  assign- 
ment whereas  other  students  may  need  to  spend°2 
hours;  the  students  may  adjust  their  own  daily 
work  schedules  accordingly.     In  this  way,  indiv- 
idual differences  in  the  rate  of  mastering  the 
material  may  be  accomodated  within  the  instructor 
paced  assignment  and  quiz  system. (Malott  and  Svin- 
icki, 1969,  p.  555) 

This  appears  to  have  all  the  advantages  of  Keller's 
system  plus  a  reduction  in  the  number  of  students  who  receive 
the  grade  of  "I".     However,  the  gain  may  be  illusory.  Hold- 
ing the  entry  and  exit  points  constant  means  that  the  addi- 
tional time  required  by  the  slow  student  must  be  taken  from 
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other  concurrent  activities;  i.e.,  social  and  recreational 
activities,  other  courses.    While  it  is  true  that  most  stu- 
dents work  at  less  than  100  percent  of  capacity,  it  is  prob- 
ably less  true  for  the  slow  student  than  for  others.  The 
student  who  has  to  work  two  hours  on  the  15  minute  assign- 
ment in  Malott  and  Svinicki's  course  will  probably  have  to 
put  extra  time  into  other  courses  too.    Robbing  Peter  to  pay 
Paul  is  a  dubious  improvement. 

Johnston  and  Pennypacker  (1970)  use  an  approach  that  is 
similar  in  its  effects.     They  permit  student-pacing  but  ar- 
range the  contingencies  in  such  a  way  that  it  is  to  the  stu- 
dent's advantage  to  take  at  least  two  performances  per  v/eek 
and  to  achieve  mastery  on  the  first  trial  for  each  unit. 
The  student  retains  his  freedom  to  respond  as  the  spirit 
moves  him;  however,  most  students  react  to  the  structure  of 
the  environment  in  the  way  the  instructor  intends.     Thus,  an 
element  of  instructor-pacing  is  superimposed  on  the  student- 
pacing  feature. 

A  Self -Contained  Classroom  Model 

In  the  present  study,  still  another  variation  is  tried. 
Students  who  are  currently  enrolled  in  the  course  provide 
proctoring  services  for  each  other.    2ach  student  in  the  ex- 
perimental group  has  a  classmate  for  a  proctor  and  is  himself 
the  proctor  for  another  student.    Unit  mastery  and  student- 
pacing  are  curtailed,  primarily  for  purposes  of  giving  the 
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experimental  treatment  a  chance  to  take  effect,  but  also  be- 
cause the  omission  of  these  features  solves  the  multiple 
exit  problem.    Actually,  this  is  simply  a  reversion  to  the 
traditional  time-based  system  in  which  multiple  exit  was  not 
a  problem.     It  ignores  the  plight  of  the  individual  student 
who  has  difficulty  with  the  material;  i.e.,  it  trades  one 
pz'oblem  for  another. 

The  proposed  model  also  introduces  problems  of  another 
kind.    Wolking  (1969)  noticed  that  students  monitoring  each 
other's  performance  showed  a  distinctive  practice  effect. 
Those  who  acted  as  proctors  prior  to  performing  typically 
received  higher  scores  than  those  who  performed  first.  When 
the  order  of  proctoring  was  reversed,  the  results  were  also 
usually  reversed.    A  question  of  importance  in  the  present 
study  is  whether  this  effect  can  be  satisfactorily  controlled. 

A  second  problem  peculiar  to  the  self-contained  class- 
room model  is  the  possibility  that  students  who  control  each 
other's  grades  will  not  adhere  to  academic  standards  as 
strictly  as  would  those  who  are  not  subjectively  involved. 
Collaboration,  whether  intentional  or  unconscious,  is  a  fac- 
tor that  must  be  taken  into  account.    Again,  the  question  is 
whether  the  effect  can  be  satisfactorily  controlled 

Restatement  of  the  Put*oose 

The  purpose  of  the  study  is  to  answer  the  following 
questions : 
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1.  Will  students  who  proctor  each  other  in  a  self- 
contained  classroom  perform  as  well  as  or  better  than  stu- 
dents who  are  proctored  by  students  who  mastered  the  course 
material  in  a  previous  term? 

2.  Will  it  be  possible  to  attain  experimental  control 
of  the  "practice  effect";  i.e.,  can  the  course  be  designed 
such  that  there  will'  be  no  particular  advantage  to  monitor- 
ing the  performance  of  a  protege  prior  to  responding  on  the 
performance  items  oneself?  Assuming  that  there  is  a  resid- 
ual practice  effect,  how  will  it  influence  the  results  ob- 
tained for  question  #1? 

3.  Will  the  performance  picture  of  those  who  proctor 
each  other  be  inflated  by  mutual  interest  and  collaboration; 
i.e.,  will  the  performance  levels  attained  in  verbal  response 
sessions,  which  are  student  proctored,  be  validated  by  the 
performance  levels  achieved  on  conventionally  monitored 
written  exams? 


CHAPTER  II 


METHOD 

Subjects 

Thirty  undergraduate  students  at  the  University  of  Flo- 
rida participated  in  the  experiment.    The  group  was  predom- 
inantly female  (two  males)  from  the  Colleges  of  Education, 
Arts  &  Sciences,  and  Health  Related  Professions.    The  first 
thirty  registrants  on  the  class  roll  were  selected  for  the 
experimental  class.     Two  of  these  were  unable  to  participate 
because  of  schedule  conflicts.    Alternates  from  an  overflow 
section  were  obtained  on  the  first  day  of  class.    All  mem- 
bers of  the  class  were  assigned  to  one  of  two  main  groups, 
those  being  proctored  by  returnees  from  a  previous  class  and 
those  proctored  by  other  currently  enrolled  students.  For 
consistency  of  identification,  these  groups  will  be  referred 
to,  respectively,  as  the  Control  group  and  the  Classroom 
group  (or  short  form,  Class  group).     The  Classroom  group  was 
further  subdivided  into  two  groups  of  seven  for  controlling 
the  order  in  which  students  received  practice  benefit  prior 
to  performing.     Reference  to  the  First  Perform  group  identi- 
fies the  group  which,  on  any  given  unit,  performed  on  the 
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adjunct  materials  prior  to  auditing  the  performance  of  the 
other  group;  the  Second  Perform  group  refers  to  those  who 
received  the  benefit  of  auditing  prior  to  performing.  The 
issue  of  practice  benefit  is  covered  in  more  detail  in  the 
section  of  this  chapter  devoted  to  that  topic. 

Figure  1  presents  the  between  groups  design  schematic- 
ally.   All  assignments  to  groups  were  made  in  accordance 
with  the  random  number  procedure  specified  by  Wyatt  and 
Bridges  H967) . 

Instructor 

Classroom  activities  were  conducted  by  the  principal 
investigator,  an  advanced  graduate  student  who  had  either 
taught  or  participated  in  the  teaching  of  the  course  on 
three  previous  occasions.     Classroom  activities  included  the 
organizational  meeting  on  the  first  day  of  class,  eighteen 
lectures,  and  the  administration  of  two  written  examinations. 
Additionally,  eighteen  verbal  performance  sessions  were  timed 
and  supervised  by  the  instructor  for  the  Classroom  group. 

Proctors 

Five  undergraduate  students  served  as  proctors  for  the 
Control  group.    All  five  had  taken  a  pilot  version  of  the 
course  in  the  preceding  term  and  had  met  criterion  for  the 
grade  of  "A".     They  had  been  trained  in  the  proctoring  as- 
signment by  the  course  instructor  and  had  assisted  in  devel- 
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oping  the  curricular  materials  taken  from  the  textbook.  The 
lecture  materials,  however,  were  not  presented  during  the 
pilot  course.     Consequently,  the  proctors  managed  their  stu- 
dents under  two  conditions:     (1)  with  some  measure  of  exper- 
tise for  the  textbook  units,  and  (2)  no  prior  experience  for 
the  lecture  units. 

Proctors  received  one  hour  of  academic  credit  for  each 
student  under  their  charge.    In  general,  their  duties  fol- 
lowed the  full  range  of  activities  described  in  detail  by 
Johnston  and  Pennypacker  (1970).    Supervisory  assistance  to 
the  proctors  was  available  from  the  instructor  before  and  af- 
ter the  timed  performance  samples.    Grades  were  based  on  the 
instructor's  evaluation  of  the  proctor's  performance  with 
their  students,  and  their  contributions  during  individual 
case  discussions  and  at  weekly  meetings. 

The  currently  enrolled  proctors—that  is,  all  14  mem- 
bers of  the  Classroom  group — received  very  little  formal 
training  in  the  proctoring  assignment.    An  explanation  of  the 
operation  of  the  system  was  given  on  the  first  day  of  class, 
and  each  student  received  a  copy  of  the  Student  Handbook,  in- 
cluding the  rales  governing  performance  sessions  (Rules  of 
thjR_Game,  Appendix  A).     In  contrast  to  this,  the  previously 
trained  students  had  received  fairly  extensive  training  dur- 
ing the  pilot  project.     The  instructor,  while  acting  as  proc- 
tor for  the  five  previously  trained  students,  had  demonstrat- 
ed the  use  of  verbal  praise  for  correct  responses,  pre-session 
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review  of  important  points  in  the  study  unit,  post-session 
tutoring,  rephrasing  of  questions  for  discussion  purposes, 
etc.     In  other  words,  the  previously  trained  proctors  had 
been  trained  in  xhe  proctoring  assignment  as  well  as  the 
textbook  subject  matter;  for  the  currently  enrolled  proctors, 
it  was  not  feasible  tc  do  this. 

Teaching  Situation 

The  course  carried  four  hours  of  credit  and  was  sched- 
uled to  meet  daily  from  9*05  to  9:55  A.M.,  Monday  through 
Thursday.    All  students  were  invited  to  attend  the  sched- 
uled class  lectures  (Appendix  A);  however,  this  was  not  com- 
pulsory in  view  of  a  university  ruling  which  makes  class  at- 
tendance optional  for  upper  classmen.     Two  class  periods  per 
week  were  devoted  to  lectures.     The  other  two  were  used  for 
obtaining  samples  of  student  performance  on  adjunctive  auto- 
instructional  materials.     Performance  sessions  were  carried 
out  in  the  regular  classroom  for  the  Classroom  group.  The 
Control  group  met  individually  with  their  proctors  in  the 
Special  Education  Learning  Center.     A  comparison  of  the  con- 
ditions under  which  the  two  groups  performed  on  the  adjunct- 
ive materials  is  shown  in  Table  I. 

Curricular  Materials 

Curricular  materials  were  taken  from  two  basic  sources: 
(1)  the  text,  Intr^djjCjtjpji. to. .Exceptional  Children,  by  S,  A. 
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Kirk  (I962),  which  was  covered  in  entirety;  and  (2)  a  lec- 
ture series  which  emphasized  the  chronological  development 
of  the  main  treatment  strategies  in  special  education.  Book 
units  were  scheduled  to  conform  to  the  chronology  of  the  lec 
ture  series  (see  schedule,  Appendix  A).     Supplemental  mater- 
ials of  the  adjunct  auto- instructional  kind  made  up  the  re- 
mainder of  the  curricular  materials. 

Adjunct  auto-instructional  materials 

Adjunct  materials  v/ere  developed  during  a  pilot  render- 
ing of  the  experimental  course  in  the  Fall  quarter,  1969. 
Six  volunteer  students  were  tested  twice  weekly  on  items 
constructed  by  the  principal  investigator.     Items  were  modi- 
fied or  discarded  for  reasons  of  ambiguity  or  vagueness  out 
not  simply  because  they  were  frequently  missed.     If  the  item 
covered  an  important  point,  it  was  retained.     The  primary 
objective  of  the  adjunct  materials  was  to  instruct,  not  sim- 
ply to  test. 

Minimizing  Practice  Effect  through  Alternate  Test  Forms 
The  practice  effect  that  was  observed  in  Wolking's 
class  has  been  described  above  (p.  17).    Practice  in  this 
context  means  auditing  another  student's  performance  ses- 
sion, then  using  the  same  test  items  on  one's  own  perform- 
ance session.     To  minimize  this  effect  in  the  present  study, 
alternate  test  forms  were  constructed  for  each  unit,  Color 
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and  number  code's  distinguished  the  alternate  forms;  thus,  a 
student  who  audited  a  performance  on  blue,  even-numbered 
cards  would  use  the  green,  odd-numbered  cards  when  his  own 
turn  came  to  perform.    Performance  sessions  were  scheduled 
so  that  any  residual  practice  benefit  was  distributed  evenly 
between  groups,  each  group  receiving  practice  (audit  first, 
perform  second)  on" the"  same  number  of  book  and  lecture  units 
as  the  other. 

Item  Length  and  Adjustment  Factors 

One  advantage  of  the  system  used  in  this  study  is  that 
it  provides  a  means  of  analyzing  some  of  the  components  of 
performance.    A  student's  rate  of  correct  responding  may  be 
influenced  by  his  reading  rate,  the  length  of  his  response 
latency,  the  rate  at  which  he  responds  incorrectly,  or  any 
of  these  in  combination.     These  categories  are  exhaustive, 
mutually  exclusive,  easily,  observed  and  measured.    As  such, 
they  provide  a  useful  monitor  on  the  system  itself  as  well 
as  on  individual  performance. 

During  the  pilot  program,  correct  responding  was  found 
to  be  closely  related  to  reading  rate.     This  in  turn  was 
negatively  correlated  with  the  mean  length  of  items  from 
unit  to  unit.     It  had  been  assumed  that  the  laws  of  chance 
would  provide  the  pilot  students  with  a  task  that  would  be 
roughly  equal  across  units,  at  least  insofar  as  the  reading 
requirement  was  concerned.     The  listing  in  Table  II  indicates 
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TABLE  II 

Derivation  of  adjustment  factors  and  adjustment  of  mean 
correct  rates  in  1?  pilot  performance  sessions 
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that  the  assumption  was  not  supported  "by  actual  measurement 
of  the  test  items. 

There  were  two  junctures  at  which  the  items  varied:  The 
length  of  the  item  (number  of  typewritten  lines  to  read);  and 
the  number  of  possible  points  (number  of  blank  spaces  per 
item).     Column  A  of  Table  II  shows  mean  lines  per  blank 
(LP3)  to  be  quite  variable  in  length,  the  longest  unit  being 
roughly  twice  the  length  of  the  shortest.    Column  C  shows 
the  mean  rate  correct  for  the  six  pilot  students.     The  high- 
ly reliable  relationship  (  r  =  -.75,  df  =  14,  P  <  .01)  be- 
tween rate  correct  and  LPB  is  illustrated  in  Figure  2,  For 
this  set  of  items,  approximately  half  of  the  variance  in  rate 
correct  may  "be  attributed  to  item  length. 

To  neutralize  the  effect  of  variant  item  length,  ad- 
justment factors  were  established  by  dividing  LPB  for  each 
unit  by  the  overall  LPB  (Table  II,  Column  B).     The  differ- 
ence between  the  functions  described  by  adjusted  and  unad- 
justed rates  is  shown  in  Figure  3.    With  the  pilot  materials, 
the  differences  were  great  enough  to  raise  questions  as  to 
the  accuracy  of  the  feedback  that  the  students  received  from 
unit  to  unit.     On  this  basis,  the  materials  were  revised  to 
reduce  item  length  variance.     This  was  accomplished  by  using 
only  one  blank  to  the  item  and  rewriting  items  that  approach- 
ed extremes  in  length.    The  materials  actually  used  in  the 
experiment  are  described  in  Table  III.    A  methodological 
question  that  the  study  attempts  to  answer  is  whether  the 
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PERFORMANCE  UNITS 


FIGURE  2 


Relationship  of  response  rates  to  length  of  items  over 
17  pilot  performance  samples,  expressed  in  mean  rate 
correct  (n  =  6)  versus  mean  lines  per  blank 
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FIGURE  3 

Comparison  of  the  rate  correct  functions  described 
by  adjusted  versus  unadjusted  scores  in  1?  pilot 
performance  sessions 
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PABLE  III 


Mean  lines  per  blank  (LPB)  and  adjustment  factors  for 
adjunct  auto-instructional  materials  used  in  this  study 
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adjustments  made  in  materials  were  successful  in  reducing 
the  relationship  between  item  length  and  performance  mea- 
sures.   This  is  taken  up  in  Appendix  C, 

Performance  Graphs  and  Criterion 

The  Behavioral  Approach  fashioned  by  Johnston  and  Pen- 
nypacker  (1970)  utilizes  two  criterion  lines  in  specifying 
the  performance  objective.     The  sample  record  in  Figure  k 
illustrates  their  method  of  graphing  performance  results. 
The  abscissa  represents  time  by  days  of  the  week.    The  or- 
dinate represents  rate  of  performance.    The  uppermost  diagon- 
al is  the  criterion  line  for  rate  correct;  the  lower  is  cri- 
terion for  rate  incorrect.     In  order  to  meet  criterion,  the 
student  must  perform  at  a  rate  which  places  the  cumulative 
performance  lines  (jagged  lines)  above  the  rate  correct  cri- 
terion line  and  below  the  rate  incorrect  criterion  line. 

In  the  present  study,  the  graphing  of  results  is  re- 
tained for  purposes  of  providing  continuous  and  cumulative 
feedbacks  however,  the  form  of  the  graph  is  changed.  Since 
performances  were  instructor-paced  rather  then  student-paced, 
the  individual  variations  in  frequency  of  performance  were 
not  of  interest  in  the  present  study.    Therefore,  it  was 
appropriate  to  convert  the  abscissa  to  a  line  representing 
units  rather  than  days  in  the  week.     This  done,  there  was 
no  further  need  to  retain  diagonal  criterion  lines.    A  diag- 
onal display  of  performance  is  desirable  in  a  free  operant 
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Illustrative  example  of  performance  graphing 
procedure  used  by  Johnston  and  Pennypacker 
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situation  because  of  the  meaningful  Skinnerian  cumulative 
record  it  produces.     In  the  present  study,  the  students  were 
not  free  to  respond  because  of  the  fixity  of  the  schedule; 
hence,  any  such  record  would  have  been  meaningless.    A  graph- 
ing procedure  of  the  kind  shown  in  Figure  5  v/as  used  instead. 
Note  that  criterion  lines  were  not  entered  on  the  graph  un- 
til after  the  tenth  performance  session.    During  the  first 
half  of  the  course,  students  were  given  feedback  only  on 
their  position  relative  to  the  class  mean.    Absolute  criter- 
ion lines  for  letter  grades  were  not  established  until  some 
measure  of  the  class's  performance  was  available  to  help  de- 
termine what  those  criteria  should  be.     This  was  recognized 
as  a  retreat  from  the  preferred  method  of  stating  criterion 
in  terms  of  some  externally  specified  behavioral  objective, 
yet  was  considered  necessary  because  the  system  did  not  per- 
mit students  to  repeat  units  to  achieve  mastery.    An  object- 
ive of  complete  mastery  is  appropriate  only  if  students  are 
provided  the  opportunity  to  reach  it. 

Figure  5  is  a  reproduction  of  the  graphic  record  of  Stu- 
dent 24.    The  two  performance  lines  represent  adjusted  dif- 
ference rates  for  individual  units  (solid  line)  and  mean  cum- 
ulative performance  (broken  line).    At  the  second  data  point, 
the  broken  line  represents  the  mean  of  performances  1  and  2; 
at  the  third,  the  mean  of  performances  1,  2  and  3;  and  so  on. 
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The  Use  of  Adjusted  Difference  Scores 

The  use  of  twin  performance  criteria  (rate  correct  and 
rate  incorrect)  poses  problems.     First,  the  nature  of  the 
feedback  may  be  ambiguous.    A  student  who  finds  himself  well 
above  criterion  for  rate  correct  but  not  below  criterion  for 
incorrect  rate  receives  equivocal  feedback.    The  reinforcing 
properties  of  the  feedback  may  be  vitiated  by  the  complexity 
of  the  information  that  is  provided.    Second,  the  system  en- 
courages suspension  of  performance  in  some  cases.     The  stu- 
dent who  is  well  above  correct  criterion  but  has  not  met  in- 
correct criterion  is  well  advised  to  stop  responding.  The 
easiest  way  to  make  no  errors  is  to  make  no  responses.  Third, 
the  system  can  be  punitive  for  high  rates  of  responding. 
Consider  the  case  of  the  student  who  reads  20  cards  in  five 
minutes.    Answering  18  correctly  and  missing  only  two,  he 
will  achieve  rates  of  3.6  correct  and  0.4  incorrect.  If 
criteria  for  rates  correct  and  incorrect  are  3.5  and  0.4, 
respectively,  criterion  will  have  been  met  in  both  cases  by 
this  student.     Compare  this  with  the  performance  of  the  stu- 
dent who  reads  and  answers  the  items  more  briskly.     Ke  goes 
through  30  items  in  the  time  it  took  the  first  student  to  go 
through  20,  missing  three  and  getting  27  correct.    His  ratio 
of  correct  to  incorrect  is  the  same,  9:1,  but  the  rates  he 
achieves  do  not  reach  criterion  for  the  unit.    While  his  rate 
correct  is  well  above  criterion,  his  rate  incorrect,  0.6, 
falls  on  the  wrong  side  of  the  incorrect  criterion  line 


38 

This  is  the  student  who  will  have  to  modify  his  behavior  in 
order  to  meet  criterion.    He  will  have  to  decrease  his  read- 
ing speed,  increase  his  response  latency,  or  stop  performing 
altogether  to  give  the  criterion  line  a  chance  to  catch  up 
to  his  incorrect  rate.    Experience  has  shown  that  the  con- 
tingency arrangements  are  potent  enough  to  produce  this  mod- 
ification; the  question  is  whether  the  change  itself  is  de- 
sirable . 

It  is  possible  to  overcome  these  objections  with  a  sin- 
gle criterion  measure  that  encompasses  all  the  information 
provided  by  the  reading  rate,  correct  rate,  and  incorrect 
rate.    Difference  rates  meet  this  requirement.     Reading  rate 
minus  correct  rate  equals  incorrect  rate,  and  correct  rate 
minus  incorrect  rate  equals  difference  rate.    The  difference 
rate  is  a  function  of  all  three  components  interacting.  It 
summarizes  the  interaction  in  a  single  score  that  accurately 
reflects  the  quality  and  quantity  of  performance.     This  com- 
prehensiveness makes  it  especially  valuable  for  research  pur- 
poses.    The  statistical  analyses  performed  in  this  study  are 
based  on  adjusted  difference  rates,  except  where  inappropri- 
ate.   The  exceptions  are  clearly  noted. 

Before  leaving  this  topic,  it  is  necessary  to  clarify 
one  further  point.     The  reference  to  reading  rates  has  been 
convenient  in  the  preceding  discussion  but  is  actually  a 
misnomer.    The  forms  exhibited  in  Figure  6  show  the  term 
"Card  Hate"  in  its  place.    This  is  more  technically  correct, 
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for  it  states  the  rate  in  number  of  cards  per  minute  that 
the  student  attempted.     "Reading  rate"  implies  reading  only, 
while  card  rate  includes  everything  that  the  student  does 
from  start  to  finish  of  the  timed  performance.    While  read- 
ing time  is  a  substantial  part  of  this,  it  is  not  the  whole. 
Response  latency  is  another  major  subdivision  of  card  rate. 
It  would  be  interesting  to  observe  these  variables  and  re- 
cord them  precisely,  but  it  has  not  been  within  the  scope  of 
this  study  to  do  so. 

Figure  6  illustrates  the  bookkeeping  method  that  was 
employed  in  the  conversion  of  raw  scores  to  adjusted  differ- 
ence scores.     The  complete  forms  from  which  these  sample 
lines  were  taken  may  be  seen  in  Appendix  B ,  along  with  a 
more  detailed  discussion  of  the  bookkeeping  method. 

Construction  of  Written  Review  Examinations 

The  two  written  review  exams  were  each  composed  of  12k 
items  randomly  selected  from  book  and  lecture  units.  The 
midterm  exam  used  items  from  the  first  eight  units;  the 
final  exam,  from  the  remaining  eight.    Each  unit  was  repre- 
sented in  the  exam  proportionately.    The  random  number  pro- 
cedure of  Wyatt  and  Bridges  (196?)  was  followed,  except  that 
duplicates  (items  that  covered  the  same  point  in  roughly  the 
same  language)  were  not  allowed,     Cne  of  the  two  would  be 
discarded  and  the  next  random  number  in  the  same  unit  se- 
lected in  its  place.    When  all  124  items  had  been  selected  in 
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Step  1  -  Raw  Data  Collection 


Individual  Performance  Record  -  Raw  Data 

Student      Jane  Doe  S  *\ 


Frst  Last              Totl    Nr  Nr     /Card  Rate  Rate', 

Unit,  ,Card[CardL    Time     Itms  Corr  Incr  ;  Rate  Corr  Ir.c 


Step  2  -  Computation  of  Adjusted  Rates 


■ 


Unit 


Performance  Adjustment  Sheet 
Adjustment  Factors:     Green  _(_o_dd)  1 .06  Blue  (even)  .98 


Student 

Adj 

Card 

Rate 

Rate 

Name- 

Fctr 

Rate 

Corr 

Incr 

Doe 

1.06 

x|  3.8 

3.2 

0.6 

Ray 

.93 

1 

■ 

 i 

44.03 


Step  3  -  Computation  and  Accumulation 
Difference  Rates* 


3.39 


Adjusted  Performance 

Record 

Name 

Jane 

Doe 

Unit 

Adj 
Card 

Rate 

Cum 
Card 
Rate 

Mean 
Card 
Rate 

Adj 

Rate 

Ccrr 

Cum 
Rate 
Ccrr 

Mean 
Rate 
Corr 

Adj 

Rate 

Incr 

Cum 
Rate 
Incr 

Mean 
Rate 
Incr 

Adj 
Diff 
Rate 

Cum\ 
Diff  I 
Rate] 

Bl 

4.03 

3.39 

0.64 

2.75 

i — i  

i 

 4  1 

I 

.J 


*3.39  -  0.64  =  2.75: 


FIGURE  6 

Three  step  computation  of  adjusted  difference  rates 


this  manner,  the  cards  were  shuffled  and  then  sorted  to  as- 
sure that  no  two  items  from  the  sane  unit  would  be  next  to 
each  other.    The  purpose  of  this  was  to  assure  that  no  mat- 
ter how  many  items  a  given  student  completed  in  the  30  min- 
ute time  period,  he  would  have  had  approximately  equal  ex- 
posure to  all  units. 

At  the  time  of  exam  administration,  students  were  in- 
structed to  take  each  item  in  turn.    Any  items  left  blank, 
prior  to  the  last  item  for  which  an  answer  was  given,  were 
counted  as  incorrect,    Students  completing  the  exam  prior  to 
expiration  of  the  allotted  30  minutes  were  instructed  to 
place  their  exams  and  answer  sheets  face  down  on  their  desks, 
and  raise  their  hands  to  have  their  actual  time  recorded. 

Control  of  Hawthorne  .Effect 

The  departure  from  routine  classroom  procedure  was 
striking  enough  to  make  the  experimental  nature  of  the  course 
obvious  to  the  students,    A  Hawthorne  effect  appeared  to  be 
inevitable.    Fortunately,  the  design  did  not  include  compar- 
isons between  Treatment  and  Mo  Treatment  groups.    It  was 
possible  to  control  for  Hawthorne  effect  by  maximizing  it 
for  all  students,  irrespective  of  group  assignments.  Appen- 
dix A  contains  the  statement  regarding  research  intentions 
that  was  given  to  all  students  the  first  day  of  class  (Notes 
to  the  Student,  in  Student  Handbook). 
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Research  Strategy 

Schutz  and  Baker  (1968)  have  discussed  the  merits  of  a 
subtractive  approach  to  experimentation  in  the  behavioral 
sciences.     The  strategy  is  distinguished  from  the  tradition 
al  additive  method  that  has  grown  out  of  agricultural  stat- 
istics.   In  the  additive  approach,  independent  variables 
are  added  to  a  presumably  neutral  situation,     The  effects 
of  the  addition  on  the  dependent  variable  are  observed  and 
compared  to  a  control  group  that  does  not  receive  the  ex- 
perimental treatment.     In  the  subtractive  approach,  the  em- 
phasis is  reversed,    A  multivariate  effect  is  produced  at 
the  starting  point  and  the  effect  of  subtracting  components 
Of  interest  is  observed.    By  process  of  elimination,  it  may 
be  possible  to  learn  which  elements  of  a  given  multivariate 
effect  are  critical  and  which  ones  superfluous, 

In  the  present  study,  the  starting  point  is  a  multivar 
iate  amalgamation  of  components  from  Keller's  Personalized 
Instruction  method  and  the  Behavioral  Approach  of  Johnston 
and  Pennypacker.    Student  performance  on  small  units  of  mat 
erial  is  recorded  directly  and  continuously,  feedback  on  pe 
formance  is  given  immediately  and  displayed  in  relation  to  i 
terminal  goal,  and  tutorial  assistance  is  available  at  the 
time  of  performance.    Admittedly,  the  important  elements  of 
unit  mastery  and  self-pacing  are  withheld  from  the  present 
model,     ks  previously  noted,  the  effect  of  these  components 
is  to  produce  a  uniform  level  of  performance  in  all  student: 
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An  experiment  that  purports  to  study  performance  differences 
cannot  accommodate  movement-based  procedures  that  obviate 
those  differences. 

Through  the  sub  tractive  approach,  the  previously- 
trained  student  proctor  is  replaced  by  an  untrained,  current- 
ly enrolled  student  proctor.     If  the  selection  criteria  for 
proctors  are  of  any  appreciable  effect  in  the  system,  their 
omission  should  be  reflected  in  diminished  student  perform- 
ance.   On  the  other  hand,  if  the  presumed  tutorial  benefit 
provided  by  a  previously  trained  proctor  is  in  reality  no 
greater  than  that  provided  by  an  interested  listener  (cur- 
rently enrolled  student),  then  the  performance  of  the  group 
using  currently  enrolled  proctors  should  suffer  no  perform- 
ance loss. 

There  are  sound  reasons  for  hypothesizing  that  the  lat- 
ter of  these  two  outcomes  will  occur;  that  is,  that  the  Class 
room  group  will  perform  at  rates  as  good  as  or  better  than 
the  Control  group  rates.    Foremost  among  these  is  the  prac- 
tice benefit  that  the  Classroom  group  receives  while  proc- 
toring  the  performance  of  classmates.    Despite  the  efforts 
to  control  it,  a  residual  practice  benefit  is  to  be  expected. 
Its  magnitude  and  relationship  to  specific  test  items  com- 
prise questions  which  must  be  answered  empirically — again, 
using  the  subtractive  approach.     The  omission  to  be  made  in 
this  case  is  of  the  students  who  received  practice  prior  to 
performing;  in  other  words,  these  who,  on  any  given  unit, 


performed  second.     Those  who  performed  first  make  up  the 
group  whose  conditions  of  performance  were  equivalent  to  the 
Control  group  with  respect  to  the  practice  variable,    A  com- 
parison of  the  performance  of  these  two  groups  should  clari- 
fy the  findings  of  the  principal  question.     If  the  subtrac- 
tion of  practice  results  in  a  loss  of  performance  vis-a-vis 
the  Control  group,  the  effect  of  practice  is  established. 
If  not,  it  becomes  necessary  to  look  elsewhere  for  the  sour- 
ces of  power  within  the  system. 


CHAPTER  III 


RESULTS 

The  Effect  of  Proctorin?  by  Previously  Trained  versus 
Currently  Enrolled  Proctors 

The  chief  burden  of  the  experiment  was  to  answer  the 
question  whether  students  in  a  self-contained  classroom, 
monitoring  each  other's  performance  under  group  performance 
conditions f  could  demonstrate  a  level  of  performance  that 
would  be  at  least  equivalent  to  that  which  has  been  repeat- 
edly produced  in  the  individualized  instructional  models 
that  Keller,  Johnston  and  Pennypacker,  and  others  have  used, 
The  affirmative  evidence  is  of  several  kinds: 

1.  Between  groups  comparison  on  the  16  first-time 
verbal  performance  sessions 

2.  Between  groups  comparison  on  the  two  repeat  per- 
formances (five  minute  verbal  responding  on  the  al- 
ternate form  of  the  unit  on  which  the  student  had 
performed  most  poorly) 

3.  Between  groups  comparison  on  the  two  30  minute 
written  performance  sessions 

These  are  considered  in  turn. 
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Between  gro-ras  comparison  on  the  16  first-time  verba], 
perf or.T.ance  sessions 

Visual  inspection  of  Figure  7  shows  that  Classroom 
group  performance  exceeded  Control  group  performance  at  all 
but  three  of  the  16  data  points.    The  differences  at  these 
points  are-  slight  compared  to  the  greater  differences  in 
favor  of  the  Classroom  group.    Selection  of  a  statistical 
test  to  analyze  these  results  was  governed  by  the  need  to 
test  not  only  the  differences  at  each  point  in  the  series 
of  scores,  but  also  the  cumulative  effect  of  these  differ- 
ences,   A  profile  analysis  procedure  described  by  Morrison 
(1967)  met  these  requirements.     The  first  step  was  to  test 
the  two  mean  vectors  for  parallelism.     In  the  absence  of 
parallelism,  a  group  interaction  must  be  assumed,  in  which 
case  the  profile  analysis  would  be  inappropriate.    A  tf.  of 
2.19  was  obtained  (df  15/14,  n.s.),  too  low  to  reject  the 
hypothesis  of  a  reliable  difference  between  mean  vectors? 
i.e.,  the  mean  vectors  were  found  to  be  statistically  paral- 
lel, 

A  second  test  was  conducted  to  determine  whether  the 
mean  vectors  were  at  the  same  level.     A  significant  differ- 
ence was  found  between  the  Classroom  and  Control  groups  (jjT 
=  3.69,  df  16/13,  P  <  .025).     The  performance  of  the  Class- 
room group  was  reliably  superior  to  the  Control  group, 
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Between  groups  comp arispn  on  the  two  re-peat  perf ormar.cas 

Table  IV  shows  the  results  of  t  tests  on  the  two  per- 
formance sessions  which  students  repeated.    No  reliable 
difference  was  found  in  the  first  of  the  repeated  units, 
although  the  mean  for  the  Classroom  group  was  again  higher, 
In  the  second  repeat  performance,  a  highly  reliable  differ- 
ence between  means  was  found  (t  =  2.66,  df  a  28,  P  <  .02), 
in  favor  of  the  Classroom  group. 

Between  groups  comparison  on  two  written  performances 

A  test  of  the  difference  between  means  for  the  two  JO 
minute  written  performances  revealed  no  reliable  difference 
between  Class  and  Control  groups.     Table  V  shows  that  while 
the  means  for  the  Classroom  group  were  higher  at  both  mid- 
term and  final  exams,  the  differences  were  not  reliable 
(midterm,  t  =  1.^3,  df  =  28,  n.s.;  final,  t  =  l.k6t  df  =  28, 
n.s. ) . 

Deferences  Between  0ra.l_.and  Written  Response  Modes 

For  purposes  of  analysing  the  data,  a  distinction  has 
been  made  between  the  16  first-time  oral  response  perform- 
ance sessions  and  the  four  review  tests.    The  16  oral  re- 
sponse performance  tests  were  composed  of  adjunct  materials 
which  had  not  been  previously  seen  by  the  students  while  the 
four  review  tests  were  composed  of  items  that  were  to  some 
degree  familiar.    Within  the  four  review  tests,  a  further 
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distinction  can  be  made  between  the  two  repeat  performances, 
which  were  done  in  the  oral  mode,  and  the  two  30  minute  re- 
view tests,  which  were  in  the  written  response  mode.  Fig- 
ure 8  illustrates  the  point  that  writing  takes  longer  than 
saying.     The  data  supporting  this  statement  are  presented  in 
Table  VI,  indicating  reliable  differences  in  both  pairs  of 
test  scores  (Set  I,  t  =  2.00,  df  =  58,  P  <  .05;  Set  II, 
t  =  2.06,  df  -  58,  F  <  .05).     Table  VII  shows  that  the  rate 
differences  are  not  the  result  of  a  lower  quality  of  per- 
formance in  the  written  mode.     The  percentages  of  correct 
responding  are  greater  for  written  mode  in  all  but  one  case. 
The  exception  is  Repeat  Test  II  for  the  Classroom  group. 
This  is  the  same  test  that  showed  the  highly  reliable  dif- 
ference on  Table  IV,  using  rate  as  the  unit  of  measurement. 
It  appears  that  this  test  may  have  come  under  an  atypical 
influence.     This  is  taken  up  in  the  Discussion  section,  Chap- 
ter IV. 

Tests.  Concerning;  the  Sffects  of  Practice 
If  the  effect  of  practice  were  the  principal  determin- 
ant of  the  performance  differences  reflected  in  the  16  oral 
response  sessions,  it  should  be  possible  to  demonstrate  this 
in  two  ways : 

1.    By  showing  that  the  seven  students  of  the  Practice 
group  produced  performance  rates  which  were  consistent- 
ly and  reliably  superior  to  the  rates  produced  by  the 
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seven  students  in  the  No  Practice  group. 
2.    By  showing  that  there  was  no  reliable  difference 
between  the  No  Practice  group  and  the  16  students  of 
the  Control  group,  who  also  performed  under  conditions 
of  no  practice. 

Comparison  of  the  Practice  and  No  Practice  Groups 

The  mean  vectors  of  the  Practice  and  No  Practice  groups 
are  plotted  in  Figure  9.    The  small  number  of  subjects  in 
the  two  groups  prevents  the  use  of  the  profile  analytic  pro- 
cedure used  earlier.    Sven  if  profile  analysis  were  avail- 
able, it  is  doubtful  that  the  mean  vectors  would  pass  the 
test  of  parallelism.    Seven  of  the  scores  are  in  the  unex- 
pected direction. 

A  test  of  the  consistency  with  which  the  practice  ef- 
fect produces  superior  scores  can  be  made  by  hypothesizing 
no  difference  in  the  probability  of  a  superior  score  between 
groups  (Hqs  P  =  ,5),    With  nine  superior  for  the  Practice 
group  and  seven  not  superior,  a  z  value  of  .27  is  obtained, 
insufficient  to  reject  the  null  hypothesis.    There  is  no 
evidence  for  the  test  of  consistency  of  superior  scores  as- 
sociated with  practice  effect. 

It  appears  that  the  lecture  units  offer  the  best  chance 
of  finding  reliable  differences  between  the  means  of  the 
Practice  and  No  Practice  groups.    Lectures  3,  5  and  6,  and 
Book  10  show  fairly  good  separation.     The  results  of  t  tests 
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on  the  means  for  each  unit  are  shown  in  Table  VIII.  Clearly, 
there  is  no  support  in  this  series  of  tests  for  the  notion 
that  practice  produces  reliably  superior  performance  rates. 
Only  one  of  the  tests  achieves  reliability  at  the  .05  level. 
This  is  hardly  more  than  chance  alone  would  produce  in  a 
series  of  16  trials. 

Comparison  of  the  No  Practice  and  Control  C-roiros 

The  more  powerful  profile  analysis  test  is  suitable 

v/hen  the  No  Practice  group  is  compared  to  the  Control  group 

(n  =  16).     Figure  10  shows  the  mean  vectors  to  be  roughly 

parallel.    The  test  of  parallelism  confirms  this  conclusion 
2 

(t    =  1.58,  df  =  15/7,  n.s.).     The  test  of  the  level  of  mean 
vectors  shows  quite  emphatically  that  the  seven  Classroom 
students  who  performed  without  practice  were  superior  to  the 
Control  group  at  a  high  level  of  reliability  (t2  =  10.^9, 
df  =  16/6,  P  <  .01). 

The  evidence  from  both  comparisons  (Practice  versus  No 
Practice,  No  Practice  versus  Control)  supports  the  conclusion 
that  the  reliably  higher  achievement  of  the  Classroom  group 
cannot  be  attributed  to  practice  benefit.    Practice  benefit 
in  this  case  is  defined  as  the  benefit  a  student  derives 
from  proctoring  the  performance  of  another  student  prior  to 
taking  his  own  performance  test  on  the  same  unit.    It  will 
be  recalled  that  the  experiment  was  designed  to  reduce  the 
effects  of  practice  by  having  Second  Perform  students  take 
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an  alternate  form  of  the  test.  The  evidence  leads  to  the 
conclusion  that  this  strategy  was  successful,  and  that  an 
explanation  for  the  superior  achievement  of  the --Classroom 
group  must  he  sought  elsewhere. 


Validation  of  Student-Proctored  Performance  Sessions 
Another  factor  which  might  reasonably  be  expected  to 
account  for  the  between-groups  performance  difference  is  the 
tendency  of  human  beings  to  collaborate  when  it  is  mutually 
advantageous  to  do  so.     The  experimental  design  included  two 
written  performance  samples  for  purposes  of  testing  this 
question.    These  were  taken  under  standard  examination  con- 
ditions, the  only  proctor  being  the  instructor, 

In  the  analysis  of  this  data,  the  use  of  adjusted  dif- 
ference scores  is  inappropriate.    It  has  been  shown  that  the 
written  exams  produced  reliably  lower  response  rates  than 
review  tests  taken  in  the  oral  response  mode.     It  was  also 
established  that  these  differences  in  performance  rates  were 
not  related  to  higher  frequency  of  errors.     The  mean  per- 
centage of  correct  responses  was  generally  higher  for  the 
written  mode  than  the  oral  mode.    It  appears  that  the  dif- 
ferences may  be  the  result  of  the  longer  time  it  takes  to 
write  an  answer  than  to  speak  it.     In  view  of  this  inequity, 
the  present  analysis  utilizes  percent  correct  as  the  unit  of 
measurement. 

If  members  of  the  Classroom  group  had  collaborated  to 
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report  higher  performance  rates  than  they  had  actually  earn- 
ed, evidence  of  this  should  be  seen  in  comparisons  of  oral 
response  performance  and  written  performance.  Collaboration 
was  possible  in  the  former  case,  not  possible  in  the  latter. 

Review  of  the  section  on  v/ritten  exam  construction 
(p.  38)  will  affirm  that  the  written  examinations  were  ad- 
ministered in  the  same  way  as  the  oral  performance  sessions, 
except  for  the  response  mode  and  the  duration  of  performing. 
Students  were  required  to  take  each  item  in  turn  and  could 
net  attain  a  higher  percentage  correct  by  skipping  around, 
looking  for  familiar  items.    It  might  be  noted  also  that  the 
nature  of  the  task  should  have  effectively  inhibited  any  last 
minute  attempts  to  master  material  which  had  not  indeed  been 
mastered  during  the  preceding  oral  performance  sessions. 
The  124  items  for  each  validation  exam  were  randomly  select- 
ed from  pools  of  more  than  500  constructed  response  items. 
To  cram  effectively  for  a  performance  task  of  this  magnitude 
would  have  been,  first,  exceedingly  difficult,  and,  second, 
not  very  rewarding.    The  written  exams  counted  for  no  more 
than  any  other  performance  sample  (five  percent  of  the 
grade ) , 

Based  on  these  considerations,  the  assumption  was  made 
that  v/ritten  exam  performance  would  provide  a  valid  certifi- 
cation of  prior  performance  records  if  the  written  exam 
score  was  approximately  equal  to  the  mean  of  the  preceding 
oral  response  units.     In  other  words,  if  members  of  the 
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Classroom  group  had  reported  performance,  rates  that  v/ere 
based  on  something  other  than  their  true  capability,  it 
should  be  seen  in  comparisons  of  oral  response  performance 
and  written  performance.    For  this  purpose,  it  is  necessary  . 
to  make  individual  rather  than  group  comparisons.  Group 
validation  of  the  kind  provided  above  in  Table  VII  (p,  54) 
might  obscure  the  fact  that  one  or  two  students  enjoyed  un- 
intended advantages  that  the  group  as  a  whole  did  not. 

Table  IX  shows  the  mean  percent  correct  for  the  eight 
oral  response  performance  sessions  on  which  Exam  I  was  based. 
The  second  column  shows  percent  correct  on  the  written  exam; 
the  difference  between  the  two  is  shown  in  the  third  column* 
The  same  data  is  given  for  both  the  Control  and  Classroom 
groups,  and  the  procedure  is  repeated  in  Table  X  for  the 
second  set  of  oral  response  rates  and  Exam  II, 

The  question  of  what  criterion  to  accept  for  test  valid- 
ation can  be  safely  ignored.     The  differences  in  test  per- 
formance are  very  slight,  more  often  positive  than  negative, 
and  of  greater  magnitude  in  the  positive  direction  than  the 
negative.     Only  one  set  of  scores  out  of  60  shows  a  percent- 
age loss  in  excess  of  10  percent.     This  student  is  in  the 
Control  group,  where  the  question  of  collaboration  is  not  at 
issue.     There  is  virtually  no  difference  between  groups  in 
this  comparison.     The  data  point  to  the  conclusion  that  the 
performance  differences  shown  in  previous  analyses  cannot  be 
attributed  to  collaboration  on  the  part  of  Classroom  students. 
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TABLE  IX 

Comparison  of  the  percent  correct  on  eight  oral  response 
samples  with  percent  correct  on  the  midterm  written 
review  exam,  by  Classroom  and  Control  groups 


CLASSROOM  CONTROL 


Mean 

Percent 

Gain  in 

Mean 

Percent 

Gain  in 

Oral 

G  nrr?  b  1" 

Mi  rl  "t"PT,Tr! 

A  AX  \J          A  ill 

l'*  .LU  L.C  I  ill 

Percent 

Midterm 

over 

Percent 

Midterm 

over 

Correct 

Exam 

Oral 

Correct 

Exam 

Oral 

.6927 

.7911 

+.0984 

.6849 

.7675 

+.0S26 

.9277 

.9585 

+.0308 

.8775 

.9025 

+.0250 

.7755 

.8954 

+.1199 

.7225 

.7900 

+.0675 

.9152 

.9833 

+.0681 

.8596 

.8789 

+.0193 

.66^4 

.7482 

+.O838 

.6800 

.7394 

+.0594 

.7090 

.9157 

+.2067 

.7404 

.7336 

-t0068 

.8395 

.7917 

-.0478 

.6730 

.7818 

+.1088 

.7783 

.8396 

+.0613 

774^ 

1  KJ  J.  Ut 

_   1  c.6i 
- . x 501 

.7833 

.8951 

+.1118 

.8308 

.92  73 

+.0965 

.6758 

.7744 

+.0986 

.8000 

.8333 

+.0333 

.75^0 

.7847 

+.0307 

.7761 

.8961 

+.1200 

.8923 

.9395 

+.0472 

.7505 

.9128 

+.1623 

.09^9 

.8293 

+.1344 

.6885 

.8346 

+.1461 

.8397 

.8876 

+.0479 

.7151 

.8916 

+.1765 

.4539 

.58^3 

+.1304 

.8152 

.8459 

+.0307 

SUMMARY 

Nr  Students  showing: 

Mean  Percentage  of: 

Loss  Gain 

Loss 

Gain 

Class 

1 

13 

.0478 

+.0877 

Control 

2 

14 

.081 5 

+.0899 
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TABLE  X 

Comparison  of  the  percent  correct  on  eight  oral  response 
samples  with  percent  correct  on  the  final  written 
review  exam,  by  Classroom  and  Control  groups 


CLASSROOM 

CONTROL 

Mean 

Percent 

Gain  in 

Mean 

Percent 

Gain  in 

Oral 

Correct 

Final 

Cral 

Correct 

Final 

Percent 

Final 

over 

Percent 

Final 

over 

Correct 

Exam 

Oral 

Correct 

Exam 

Oral 

.7828 

.7754 

Q 1  a  0 
,0107 

+  .0/.55 

.9^74 

.9843 

.9193 

r\  0  /**.  r\ 

.92'jO 

+.0007 

.8333 

.8  546 

.7336 

-,0922 

,9619 

.9848 

1  noon 

.7920 

.7748 

-.0172 

.7320 

.6900 

.7375 

Q  h  nh 

,0474 

+ . 0899 

.7305 

.8800 

.7500 

+. o^^3 

.8727 

.8463 

-.0264 

6576 

+  1  ni  7 

.8376 

.8058 

-.0318 

.6495 

.5583 

-.0912 

.8382 

.8958 

+.0576 

.8007 

,9200 

+.1193 

.8O5O 

.7897 

-.0153 

.8789 

.8140 

- , 0649 

.8945 

+.0910 

.7362 

.8147 

+,0785 

.9000 

.9683 

+.0674 

.8426 

.9356 

+.0930 

.6994 

.7467 

+.0473 

.8142 

.7484 

-.0658 

.7573 

.7506 

-.0067 

.  74,22 

.8107 

+.0685 

.5138 

.6666 

+.1528 

.8539 

.7748 

-.0791 

SUMMARY 

Nr  Students  Showing: 

Mean  Percentage  of: 

Loss  Gain 

Loss 

Gain 

Class 

6 

8 

.0216 

+.0606 

Control 

6 

10 

.0684 

+.0805 
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Proctor  Training  and  Student  Performance 
Prom  the  foregoing  it  might  appear  that  the  Control 
group  proctors  contributed  nothing  of  consequence  to  student 
performance.    Such  a  conclusion  is  not  dictated  "by  the  data. 
What  the  study  shows  is  that  the  performance  of  the  Class- 
room group  was  superior  xo  that  of  the  Control  group,  and 
that  neither  the  effects  of  practice  nor  collaboration  ac- 
count for  this  difference.    No  warrant  is  provided  for  as- 
suming that  previously  trained  proctors  contribute  nothing 
to  student  performance.     It  would  be  as  reasonable  to  assume 
that  currently  enrolled  proctors  contribute  as  much  to  their 
students  as  their  previously  trained  counterparts  do. 

A  measure  of  this  variable,  albeit  a  weak  one,  is 
available  within  the  experimental  design.    Proctoring  for 
the  Control  group  was  carried  out  under  two  levels  of  proc- 
tor sophistication — a  fairly  high  level  of  expertise  for  the 
book  units,  and  no  prior  experience  with  material  in  the  lec- 
ture units.    Assuming  that  the  Control  group  proctors  gave 
substantial  tutorial  assistance  to  their  students  on  the  book 
units  and  almost  none  on  the  lecture  units,  it  follows  that 
the  proportion  of  incorrect  lecture  items  on  the  written  re- 
view test  should  be  higher  than  the  proportion  for  book  items. 
Unfortunately,  a  test  of  this  kind  does  not  take  into  account 
the  possibility  of  unequal  levels  of  difficulty  between  lec- 
ture and  book  units.     If  book  units  are  generally  the  more 
difficult  of  the  two,  the  hypothesized  effect  would  tend  to 
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be  cancelled  out;  conversely,  if  lecture  units  were  more 
difficult,  the  effect  would  be  exaggerated.    A  test  is  still 
possible,  however,  if  the  classroom  group  is  used  as  a  con- 
trol.   The  tutorial  benefit  provided  by  Classroom  proctors 
would  be  based  on  equal  familiarity  with  lecture  and  book 
units  alike;  hence,  free  of  the  systematic  variance  which  is 
presumed  to  be  present  in  the  Control  group. 

Table  XI  shows  chi  square  contingency  tables  for  the 
first  and  second  review  examinations.    Expected  frequencies 
for  the  null  hypothesis  are  shown  in  the  cell  insets  and 
observed  frequencies  in  the  cells  proper.    For  the  first 
test,  a  chi  square  of  ^.632  is  obtained,  sufficient  to  re- 
ject the  null  hypothesis  (P  <  .05).    In  the  second  test,  the' 
differences  between  expected  and  observed  frequencies  are 
negligible  (X2  =  .102,  df  =  1,  n.s.). 

Review  of  Results 

1.  Classroom  group  performance  in  the  sixteen  oral  re- 
sponse sessions  was  superior  to  Control  group  performance  at 
a  moderately  high  level  of  reliability  (P  <  .025). 

2.  On  the  re\'iew  tests,  the  mean  performance  score  for 
the  Classroom  group  was  higher  in  four  out  of  four  cases; 
however,  only  one  of  these  differences  was  reliable  using 
individual  t  tests.     This  was  the  second  test  of  oral  re- 
sponding, in  which  the  Classroom  group  was  reliably  higher 
(P  <  .02). 
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3.  In  an  effort;  to  rule  out  undesirable  influences 
that  might  account  for  the  above  results,  two  possibilities 
were  evaluated,  the  effects  of  practice  and  the  effects  of 
collaboration: 

a.  Evidence  that  the  experimental  design  success- 
fully controlled  the  effects  of  practice  was  cf  two 
kinds.    First,  there  was  no  appreciable  difference 
between  the  performance  of  the  Practice  group  and  the 
No  Practice  group.     Second,  the  performance  of  the  No 
Practice  group  was  superior  to  the  Control  group  at  a 
high  level  of  reliability  (P  <  .01). 

b.  Successful  control  of  possible  student  collab- 
oration was  controlled  through  an  inspection  of  dif- 
ferences between  oral  and  written  performance.  Col- 
laboration was  possible  for  the  Classroom  group  in 
one  condition  but  not  the  other.    A  comparison  of  the 
Classroom  and  Control  groups  showed  the  two  to  be  vir- 
tually identical. 

4,  Qualified  support  was  found  for  the  hypothesis  that 
Control  group  students  would  not  do  as  well  with  lecture 
items  on  xhe  review  tests  as  Classroom  students.     The  under- 
lying supposition  v/as  that  the  Control  proctors,  having  had 
no  experience  with  the  lecture  materials,  would  provide  poor 
tutorial  assistance  for  the  lecture  units,  and  that  this 
would  be  reflected  in  the  written  exam  results,    A  reliable 
difference     (P  <  .05)  v/as  observed  for  data  from  the  midterm 
exam  but  not  for  the  final.  , 


CHAPTER  IV 


DISCUSSION 

The  purpose  of  this  study  was  to  evaluate  one  of  the 
principle  components  of  an  instructional  system  that  has 
shown  exceptional  merit  in  college  classrooms  (Keller,  1968a 
McMichael  and  Corey,  I9691  Johnston  and  Pennypacker,  1970). 
The  component  of  interest  is  the  proctoring  arrangement. 
This  has  been  generally  accepted  as  one  of  the  cornerstones 
of  the  personalized  instruction  method  and  has  been  the  sub- 
ject of  an  extraordinary  volume  of  descriptive  comment.  The 
eulogizing  of  the  proctor  has  focused  on  his  tutorial  skills 
his  competence  in  handling  a  variety  of  student  problems, 
and  his  capacity  to  engage  peers  in  the  kind  of  interperson- 
al relationship  that  is  seldom  possible  for  the  instructor 
of  a  large  class.     Diagnosis  and  remediation  are  within  his 
range;  individualized  instruction,  through  timely  curricu- 
lum modification,  appears  to  be  part  of  his  routine  (cf., 
Johnston  and  Pennypacker,  1970). 

Yet  the  proctor  differs  from  the  currently  enrolled 
student  mainly  by  virtue  of  his  having  completed  the  course 
a  term  or  two  earlier.    Any  qualities  he  might  have  achieved 
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beyond  that  single  distinction  might  as  readily  be  found  in 
the  currently  enrolled  student.     It  has  been  the  thesis  of 
this  study  that  if  a  previously  trained  peer  can  effectively 
participate  in  the  training  of  a  college  student,  a  current- 
ly enrolled  classmate  can,  too;  that  the  amount  of  learning 
accomplished  by  the  currently  enrolled  student  is  only  weak- 
ly related  to  the  proctor's  command  of  the  subject  matter, 
if  at  all;  and  that  what  the  currently  enrolled  student 
loses  by  not  having  a  previously  trained  proctor,  he  recov- 
ers through  his  own  experience  of  proctoring  another.  These 
points  are  discussed  in  the  light  of  the  experimental  evi- 
dence . 

lhe_£ffectiveness  of  ,a  System  that  Uses  Currently  Enrolled 
Classmates  as  Proctors 

Perhaps  the  strongest  result  in  the  study  is  the  reli- 
able difference  between  the  group  proctored  by  previously 
trained  students  and  the  group  proctored  by  currently  en- 
rolled classmates.     In  17  of  20  performance  measures,  the 
Classroom  students  achieved  higher  performance  rates  than 
the  students  who  were  proctored  by  "experts"  (those  who  had 
previously  demonstrated  a  high  degree  of  competence  with  the 
material).    With  the  possible  effects  of  practice  and  col- 
laboration ruled  out  as  sources  of  this  difference,  the  var- 
iance may  be  assumed  to  lie  within  the  different  treatments. 
However,  there  is  no  justification  for  concluding  that  the 
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currently  enrolled  student  is  a  more  effective  tutor  than 
the  previously  trained  student.     The  two  proctoring  systems 
include  differences  other  than  the  tutorial  assistance  var- 
iable (see  Table  I,  p.  23),  and  the  experiment  does  not  ade- 
quately test  the  components  operating  within  the  system, 
only  the  global  effects  of  the  systems  themselves. 

The  stated  purpose  of  the  study  was  to  evaluate  a  prag- 
matic alternative  to  an  established  proctoring  method.  To 
meet  its  objective,  it  need  only  show  that  a  proctoring  sys- 
tem utilizing  currently  enrolled  students  as  proctors  can  be 
as  effective  as  a  system  that  uses  previously  trained  stu- 
dents.    This  objective  has  been  met.     The  evidence  is  con- 
sistent across  a  relatively  large  number  of  performance  sam- 
ples and  is  reliable. 

The  Proctor  Training  Variable 

As  noted  above,  the  experiment  does  not  provide  an  ade- 
quate test  of  the  question  whether  a  proctor's  prior  train- 
ing in  the  subject  matter  favorably  influences  the  perform- 
ance of  his  students.    However,  weak  evidence  was  found  for 
the  obverse  proposition:    That  a  proctor's  lack  of  training 
in  the  subject  matter  can  have  an  adverse  effect  on  the  per- 
formance of  his  students.     In  the  first  of  two  written  re- 
view tests,  the  Control  group  students  showed  a  significantly 
greater  proportion  of  incorrect  responses  among  lecture 
items  than  the  Classroom  group  did.     It  was  on  the  lecture 
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items  that  the  Control  group  proctors  had  received  no  prior 
training;  and,  presumably,  these  items  on  which  they  would 
have  been  able  to  give  their  students  no  tutorial  assist- 
ance . 

Had  the  same  result  been  observed  on  the  final  exam, 
the  finding  could  be  stated  with  greater  conviction.  A 
possible  explanation  of  the  discrepancy  is  that  the  novelty 
of  the  proctoring  arrangement  had  diminished  by  the  second 
half  of  the  course,  thereby  decreasing  its  tutorial  effect- 
iveness.   A  decrease  in  tutorial  effectiveness  would  have 
reduced  the  bias  favoring  book  items.    This  in  turn  would 
result  in  the  nearly  equal  proportions  that  were  found  on 
the  final  exam.    Diminished  tutorial  effectiveness  would 
also  account  for  the  slightly  greater  performance  loss  that 
the  class  experienced  in  the  final  exam  (cf.,  Table  IX  and 
Table  X). 

In  any  event,  the  evidence  on  this  point  is  inconclu- 
sive.   A  good  test  could  be  made  by  adding  completely  naive 
proctors  to  the  model  which  uses  previously  trained  and  cur- 
rently enrolled  proctors.     If  the  other  conditions  of  the 
performance  session  were  held  constant,  between-group  dif- 
ferences cculd  be  more  closely  tied  to  the  proctor  training 
variable.     It  should  not  be  surprising  to  find  no  difference 
at  all.     One  is  hard  pressed  to  find  a  consistent  relation- 
ship between  student  performance  and  the  sophistication  of 
the  instructor;  why  should  it  be  expected  between  proctor 
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and  student?    The  facilitation  of  performance  appears  to  be 
a  complex  phenomenon,  to  say  the  least, 

Ferster  (1968)  puts  the  question  of  proctor  sophistica- 
tion In  perspective.    In  the  course  he  has  reported,  it  is 
the  function  of  the  responder  to  respond  and  the  listener  to 
listen.     One  might  assume  that  the  responder  learns  to  re- 
spond by  responding.    The  presence  of  an  interested  listener 
is  enough  to  maintain  the  behavior  for  most  students.  The 
point  is  that  one  needs  no  special  training  to  become  an  in- 
terested listener.    A  currently  enrolled  student  should  be 
able  to  do  this  as  well  as  anyone  else.    Certainly  he  has 
the  incentive. 

Another  point  should  be  made  before  leaving  this  topic. 
The  proctor  training  variable  has  been  dichotomized,  between 
those  who  were  previously  trained  and  those  who  were  current- 
ly enrolled.     This  does  not  mean  that  the  currently  enrolled 
students  were  untrained  vis-a-vis  the  previously  trained 
proctors.    The  Classroom  group  proctor  was  required  to  pre- 
pare for  the  performance  sample  in  just  the  way  that  his 
protege"  was.    What  he  lacked  in  depth  of  experience,  com- 
pared to  the  Control  group  proctor,  he  probably  made  up  in 
the  immediacy  of  his  contact  with  the  material. 

In  sum,  the  net  effect  of  the  proctor  training  vari- 
able is  thought  to  have  been  very  slight  in  the  present  ex- 
periment.    It  is  necessary  to  look  elsewhere  for  the  vari- 
ables that  made  the  difference. 
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The  Proctorin."-  Role  as  a  Strengthener  of  Student 
Performance 

It  has  been  understood  from  the  beginning  (Keller, 
1968a)  that  a  proctor's  duties  strengthened  his  own  learning 
as  well  as  that  of  his  protege.    However,  the  contingencies 
are  not  arranged  to  assure  this  result.    What  the  previously 
trained  proctor  learns  in  performance  sessions  is  almost  in- 
cidental to  his  true  purpose,  more  unavoidable  than  planned. 
This  contrasts  sharply  with  the  currently  enrolled  proctor. 
He  goes  into  the  proctcring  assignment  under  an  altogether 
different  sot  of  contingencies.    He  knows  that  some  of  the 
material  he  listens  to  will  appear  on  a  review  test  he  will 
take.    Many  of  the  items  he  hears  will  confirm  or  clarify 
his  understanding  of  the  material  he  has  prepared  for  his 
own  performance  session.    He  has  an  incentive  to  listen 
thoughtfully  to  the  performance  of  his  protege',  even  though 
his  own  performance  test  will  be  composed  of  a  different  set 
of  items. 

These  contingencies  remain  in  effect  not  only  during 
the  timed  portion  of  the  performance  session  but  throughout 
the  entire  50  minute  period.     The  discussion  is  maintained 
in  strength  because  everyone  has  an  interest  in  maintaining 
it,     The  coverage  and  exchange  of  information  is  further 
enhanced  by  the  rotation  of  students  for  the  second  perform- 
ance sample.    If  Student  A  is  proctored  by  Student  3  in  the 
first  sample,  he  will  be  paired  with  Student  C  for  the  sec- 
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ond.    Meanwhile,  Student  3  will  be  taking  his  performance 
test  with  Student  D.    Both  C  and  D  will  have  brought  to 
their  discussions  the  information  and  ideas  that  they  ex- 
changed with  Students  E  and  ?  during  the  first  half  of  the 
performance  period. 

It  is  in  the  redundancy  of  the  arrangement,  in  the 
overlapping  and  inseparability  of  roles,  that  a  powerful 
opportunity  for  strengthening  performance  appears  to  reside. 
The  contingencies  are  arranged  to  make  the  most  of  it.  Each 
student  is  reinforced  both  as  speaker  and  listener.     One  may 
speculate  that  the  group  pools  its  resources  and  cross- 
fertilizes  its  members  in  a  way  that  is  not  possible  for 
students  who  keep  appointments  for  individualized  instruc- 
tion. 

Unfortunately,  the  experimental  design  is  silent  on 
this  question.     The  foregoing  description  is  not  based  on 
experimental  evidence  but  on  observation  of  how  the  system 
worked  once  it  was  set  in  motion.     This  was  unknown  prior 
to  the  time  it  was  tried;  indeed,  it  was  an  empirical  ques- 
tion whether  it  would  work  at  all.     The  experimental  results 
should  be  clarified  in  respect  to  one  point,  however.  It 
has  been  shown  that  the  practice  effect  was  successfully 
controlled  in  this  experiment,  yet  the  suggestion  is  now 
made  that  it  was  precisely  the  additional  discussion,  ex- 
change of  information  between  students — i.e.,  "practice"  

that  accounts  for  the  superior  performance  rates  of  the 
Classroom  group. 
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To  resolve  this  apparent  contradiction,  it  is  necessary 
to  make  a  distinction  between  desirable  and  undesirable  prac- 
tice.   In  the  former  case,  the  practice  is  based  on  the  stu- 
dent's preparation  of  the  assignment  (reading  the  book, 
studying  the  lecture  notes,  etc.).     It  is  just  this  kind  of 
practice  that  the  course  is  designed  to  maximize.     It  was 
the  practice  on  specific  test  items  that  had  to  be  controlled 
in  the  Classroom  model.    Keller  comments  on  this  distinction 
in  the  following  passage,  part  of  which  was  quoted  in  the 
introductory  chapter; 

The  'response'  is  not  simply  the  completion  of 
a  prepared  statement  through  the  insertion  of  a 
word  or  phrase.    Rather,  it  may  be  thought  of  as 
the  resultant  of  many  such  responses,  better  de- 
scribed as  the  understanding  of  a  principle,  a 
formula,  or  a  concept,  or  the  ability  to" use  an 
experimental  technique.    Advance  within  the  pro- 
gram depends  on  something  more  than  the  appear- 
ance of  a  confirming  word  or  the  presentation  of 
a  new  frame  ....     (Keller,  1968a,  p.  84) 

In  other  words,  performance  on  the  test  items  represents  an 
understanding  of  the  material,  not  simply  an  ability  to  mem- 
orize responses  to  stimulus  items.    If  this  were  not  so,  it 
would  be  satisfactory  to  distribute  copies  of  the  one  thous- 
and or  more  items  and  let  the  students  commit  them  to  mem- 
ory. 

The  test  of  practice  effect  consisted  of  demonstrating, 
first,  that  students  who  proctored  first  and  then  took  their 
performance  tests  did  not  show  significantly  better  perform- 
ance; and,  second,  that  the  margin  of  superiority  over  Con- 
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trol  group  students  was  maintained  even  when  the  group  that 
could  have  "benefitted  from  practice  (the  Second  Perform 
group)  was  removed  from  the  comparison.    Attacking  this  ques- 
tion from  different  fronts  not  only  makes  the  point  emphat- 
ically but  also  yields  information  with  different  shades  of 
meaning.    In  the  first  instance,  it  is  shown  That  students 
will  not  always  do  better  simply  by  listening  first  to  an- 
other student's  performances  in  the  second,  it  is  shown  that 
the  variables  that  produced  a  reliable  difference  between 
Control  and  Classroom  groups  were  at  work  during  the  entire 
performance  period,  not  just  the  last  25  minutes  of  it  (when 
the  Second  Perform  group  took  the  performance  test). 

This  is  where  the  data  stops— close  to  an  answer  but 
incapable  of  providing  it.    It  might  be  possible  to  answer 
it  conclusively  by  setting  up  still  another  treatment  group, 
one  that  received  proctoring  under  Classroom  group  condi- 
tions but  did  not  proctor  in  turn.    Comparing  a  group  such 
as  this  to  Control  and  Classroom  groups  would  tell  what  part 
of  the  variance,  if  any,  can  be  attributed  to  proctoring  as 
a  learning  experience  (strengthener  of  performance).    If  the 
No  Proctoring  group  came  up  with  scores  drawn  from  the  same 
population  as  the  Control  group,  the  hypothesized  value  of 
proctoring  as  a  learning  experience  would  be  confirmed;  if 
drawn  from  the  same  population  as  the  Classroom  group,  re- 
jected; and  if  drawn  from  a  population  somewhere  betv/een  the 
two,  it  would  be  confirmed  but  would  not  tell  the  whole  story. 
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If  this  last  alternative  were  the  case,  it  might  he 
fruitful  to  look  also  at  the  role  of  respondents  in  depres- 
sing performance.    Some  students  find  the  unfamiliar  per- 
formance session  anxiety-producing.    A  situation  that  evokes 
respondents  can  seriously  weaken  performance  (Ferster  and 
Perrott,  1968,  p.  130).     The  question  is  whether  sitting 
alone  in  an  experimental  atmosphere  with  a  previously  trained 
peer  (one  who  has  already  achieved  success  with  the  material) 
is  more  anxiety-producing  than  taking  the  performance  ses- 
sions in  the  more  familiar  classroom  along  with  others  who 
are  in  the  same  boat.    In  this  study,  at  least  three  pscple 
were  enough  bothered  by  this  factor  to  complain  of  it  open- 
ly.   Two  of  these  were  in  the  Control  group,  one  in  the 
Class, 

A  Gratuitous  Finding:     The  Effect  of  Time  on  Practice 
Benefit 

It  has  been  shown  that  the  experiment  successfully  con- 
trolled the  effects  of  practice  on  specific  stimulus  items. 
Demonstrating  that  an  effect  did  not  take  place  is  appropri- 
ate in  an  experiment  that  specifically  sets  out  to  assure 
this  result;  however,  the  conclusion  would  be  strengthened 
if  information  were  available  to  show  what  happens  when  the 
effect  is  not  controlled.     There  is  one  test  among  the  per- 
formance samples  that  may  provide  information  on  this  point. 

The  observation  was  made  earlier  (p.  50)  that  an  atyp- 
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ical  influence  appeared  to  be  at  work  for  the  Classroom 
group  in  Repeat  Test  II,     This  influence  could  very  well 
have  been  the  effect  of  practice  on  specific  stimulus  items. 
It  will  be  noted  (Figure  1,  p.  22)  that  Repeat  Test  II  was 
scheduled  at  the  midpoint  of  the  second  half  of  the  course. 
This  limited  to  four  the  number  of  performance  units  from 
which  repeat  tests  could  be  chosen.     Consequently,  the 
Classroom  group  students  would  have  listened,  within  the 
preceding  two  weeks,  to  the  same  items  on  which  they  were 
now  taking  the  performance  test.    This  would  not  be  true  for 
the  Control  group  because  of  the  requirement  that  all  stu- 
dents use  the  alternate  form  of  the  unit  on  which  they  had 
performed  most  poorly.    The  Classroom  group  students  would 
have  had  the  same  advantage  in  Repeat  Test  I;  however,  this 
test  came  at  the  end  of  the  first  half  of  the  course.  This 
meant  that  some  of  the  students  would  be  repeating  units 
that  had  not  been  seen  in  several  weeks  (by  actual  count, 
the  mean  number  of  units  separating  the  performances  for  Re- 
peat Test  I  was  ^.2;  for  Repeat  Test  II,  2.3). 

The  foregoing  is  offered  as  a  gratuitous  result.  The 
scheduling  of  Repeat  Test  II  was  not  an  experimental  manipu- 
lation but  a  matter  of  necessity.     If  the  above  interpreta- 
tion is  correct,  the  retention  of  repeat  material  across 
time  periods  might  be  an  interesting  variable  to  investi- 
gate.   This  would  be  especially  true  if  a  comparison  of  the 
two  types  of  practice  benefit  discussed  above  (specific  item 
versus  total  material)  could  be  experimentally  tested. 
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Departures  from  the  Behavioral  Model 

Ironically,  this  evaluation  of  the  behavioral  approach 
to  college  instruction  has  consisted  of  stripping  it  of  its 
uniquely  behavioral  elements.     The  objection  might  be  raised 
that  the  suspension  of  unit  mastery  and  self -pacing  so  al- 
ters the  format  of  the  personalized  instruction  model  that 
the  test  of  the  proctoring  question  cannot  produce  a  valid 
answer.    Keller  (1968b)  warned  of  the  consequences  of  chang- 
ing components  within  the  system,  stating  that  "a  change  in 
one  area  may  conceivably  have  bad  effects  in  others"  (Keller, 
1968b,  p.  11).     He  called  particular  attention  to  the  most 
distinctive  features  of  the  systems    Self-pacing,  unit  mas- 
tery, and  student  proctors.     The  present  study  has  changed 
one  of  these  and  had  to  abandon  the  other  two  in  order  to 
evaluate  the  change , 

It  is  possible  that  the  effectiveness  of  previously 
trained  proctors  cannot  be  demonstrated  unless  the  integrity 
of  the  system  is  maintained.     The  repetition  of  unmastered 
units  may  reflect  the  salutary  effects  of  tutoring  better 
than  the  two  review  exams  used  in  this  experiment.  However, 
there  is  not  much  support  for  this  view  in  data  given  by 
Johnston  and  Fennypacker  (1970).     In  their  studies,  it  was 
repeatedly  shown  that  the  mean  gain  realized  on  repeat  per- 
formances   was  modest,  especially  in  terms  of  the  reduction 
of  errors  (mean  error  reductions  of  5  percent  or  less  on 
three  studies).    Working  within  this  kind  of  margin,  it 
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would  be  exceedingly  difficult  to  demonstrate  the  benefit  of 
tutorial  assistance. 

In  any  event,  the  critical  thing  to  remember  is  that 
there  was  not  a  shred  of  evidence  anywhere  in  this  study 
that  would  demean  the  ability  of  the  previously  trained  proc 
tor.    The  evidence  does  not  refute  the  assumptions  on  which 
Keller  (1968a)  and  Johnston  and  Penny-packer  (1970)  have 
based  the  selection  of  proctors;  it  does,  however,  suggest 
that  whatever  it  is  that  proctors  do  to  facilitate  student 
performance,  the  currently  enrolled  student  can  do  as  well. 

It  appears  that  the  self-contained  class,  using  cur- 
rently enrolled  students  for  proctors,  can  be  as  effective 
as  systems  which  utilize  experienced  proctors. 

Application  Of  the  Pinrlrn/rg 

Nothing  in  the  above  should  be  taken  to  mean  that  the 
experimental  method  used  in  this  study  is  the  best  way  to 
organise  a  college  class.    Decisions  on  how  to  use  compon- 
ents of  the  system  must  be  made  individually  to  suit  local 
circumstances  and  curriculum  objectives.     For  example,  a 
distinction  was  made  earlier  between  the  behavior  principles 
subject  matter  and  the  non-sequential  material  of  the  ex- 
ceptional child  survey  course.    A  criterion  objective  of  90 
percent  correct  responding  may  be  appropriate  for  behavior 
principles,  just  as  an  accuracy  criterion  of  100  percent 
might  be  required  for  a  pharmacist  mixing  medicinal  com- 
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pounds.    Neither  of  these  cases  should  govern  the  selection 
of  criterion  for  the  exceptional  child  course.    The  nature 
of  the  subject  matter,  the  objectives  of  the  instructor,  the 
needs  of  the  individual  students — these  are  the  proper  deter 
minants  of  such  questions.     If  it  is  appropriate  to  bring  90 
percent  of  the  class  to  criterion,  unit  mastery  and  self- 
pacing  should  be  used.    However,  it  should  be  kept  in  mind 
that  unit  mastery  is  expensive  in  terms  of  the  number  of 
test  items  required  to  use  it  properly.    If  the  student  sees 
the  same  items  the  second  time  round  that  he  saw  the  first, 
he  may  be  learning  little  more  than  discrete  responses  to 
stimulus  items. 

It  is  in  view  of  considerations  such  as  these  that  the' 
method  used  in  this  study  represents  a  pragmatic  alternative 
to  established  proctoring  patterns.    The  contention  here  is 
that  the  behavioral  or  personalized  approach  to  college  in- 
struction may  be  more  flexible  than  its  application  to  date 
has  shown  it  to  be.     It  would  seem  that  the  key  to  its 
broadest  and  most  successful  use  will  not  be  found  in  a  dog- 
matic adherence  to  established  methods  but  in  a  willingness 
to  explore  the  many  possible  combinations  of  components,  and 
how  those  combinations  will  produce  a  variety  of  effects  in 
student  performance. 


CHAPTER  V 


SUMMARY  AND  CONCLUSIONS 

The  study  was  concerned  with  Keller's  (1968a)  behavior- 
al approach  to  college  instruction.    Using  Johnston  and  Pen- 
nypacker's  (1970)  verbal  response  unit  as  the  measure  of 
performance,  the  experiment  tested  the  question  whether  cur- 
rently enrolled  students  could,  as  a  pragmatic  alternative 
to  the  established  model,  fulfill  the  proctoring  role  in  a 
self-contained  class.    To  allow  a  proper  test  of  the  ques- 
tion to  be  made,  it  was  necessary  to  withhold  use  of  the 
unit  mastery  and  self -pacing  features.     The  results  were: 

1 ,  The  group  using  currently  enrolled  proctors  achiev- 
ed reliably  better  performance  results  than  the  group  using 
previously  trained  proctors, 

2,  The  effects  of  practice  on  specific  stimulus  items 
were  successfully  controlled,  ruling  out  this  undesirable 
source  of  variance  as  a  possible  cause  of  the  difference 
between  groups, 

3,  The  effects  of  possible  collaboration  between  stu- 
dents who  were  proctoring  each  other  were  ruled  out  as  a 
source  of  the  difference  between  groups. 
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^.    The  rats  of  oral  responding  on  five  minute  repeat 
performance  samples  was  found  to  "be  reliably  higher  than 
written  responding  on  30  minute  review  tests. 

5.    The  majority  of  students  achieved  higher  percent- 
ages of  correct  responses  on  the  written  review  tests  than 
on  the  eight  oral  response  performances  that  preceded  them. 
The  mean  gain  in  corx-ect  percentage  was  of  greater  magnitude 
than  the  mean  loss. 

From  the  results,  the  following  conclusions  were  drawn: 

1.  The  applicability  of  the  behavioral  approach  (Kell- 
er, 1968a;  Johnston  and  Pennypacker,  1970)  to  college  in- 
struction has  been  successfully  extended  to  another  subject 
matter  field. 

2.  Proctoring  the  performance  of  other  students  appears 
to  provide  a  powerful  opportunity  for  strengthening  perform- 
ance, particularly  for  students  who  are  reinforced  for  using 
it  as  such;  i.e.,  currently  enrolled  students. 

3.  The  combination  of  learning  as  student  proctor  and 
as  student  responder  leads  to  higher  rates  of  accurate  re- 
sponding than  does  learning  as  student  responder  alone. 

The  proctor-responder  combination  is  a  feasible 
classroom  arrangement,  and  can  be  used  without  lessening  the 
quality  of  performance  through  illegitimate  collaboration  or 
practice  on  specific  stimulus  items. 

5.    The  method  described  in  this  study  is  a  suitable 
pragmatic  alternative  to  the  Personalized  Instruction  method 
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of  Keller  (1968a)  and  the  Behavioral  Approach  of  Johnston 
and  Pennypacker  (1970),  and  may  he  used  with  or  without  the 
unit  mastery  and  self -pacing  features. 
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NOTES  TO  THE  STUDENT 
The  procedure  we  will  follow  this  quarter  has  already 
been  explained  in  class.    These  additional  comments  will  tell 
you  more  about  the  system.    We  hope  you  will  see  your  part  in 
the  course  more  clearly  and  understand  some  of  the  reasons  we 
have  set  it  up  this  way.    Bear  in  mind  that  there  is  much  to 
be  learned  about  precision  teaching  in  the  classroom.  Your 
comments,  criticisms  and  suggestions  are  earnestly  solicited. 
Feel  free  to  contact  the  instructor  at  any  time.    His  name  is 
John  Gaynor.    He  can  be  found  at  the  Learning  Center  and  Spec- 
ial Education  Department,  Room  ^3,  Norman  Hall. 

Frequent  Performance  under  Appropriate  Reinforcement 

Precision  teaching  is  a  product  of  behavior  analysis.  It 
seeks  to  apply  experimentally  derived  learning  principles  to 
the  classroom.    Although  it  is  similar  in  some  respects  to 
programed  instruction,  it  comes  closer  to  being  a  middle  step 
between  conventional  methods  and  the  teaching  machine.  The 
science  of  teaching  is  exploited  but  so  is  the  human  element. 
You  will  probably  have  more  interpersonal  contact  in  this 
course  than  you  do  in  others. 

Teachers  have  known  for  centuries  that  children  learn  by 
doing.    So  do  adults,  college  students,  pigeons  and  rats.  Yet 
much  of  our  educational  system  fails  to  provide  sufficient 
opportunity  for  the  student  to  perform  to  "do"— under  ap- 
propriate reinforcement  conditions;  that  is  to  say,  under 
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conditions  which  attach  real  consequences  to  the  performance. 
A  real  consequence  in  this  context  is  the  grade  that  is  earned 
in  the  course.     The  typical  college  course  provides  two  such 
performance  opportunities,  a  midterm  and  final  exam.     In  the 
course  you  are  about  to  take,  "doing"  occurs  more  frequently, 
twice  a  week  rather  than  twice  a  quarter,     Instead  of  two  mas- 
sive doses  of  performance,  there  are  twenty  bite-size  doses. 

One  advantage  of  having  many  small  performance  sessions 
is  that  you  get  a  lot  of  information  (sometimes  called  feed- 
back) on  where  you  stand  with  respect  to  the  grade  you  wish  to 
earn.     If  the  preparation  you  make  doesn't  get  you  the  grade 
you  want,  you  don't  have  to  wait  until  after  you've  bombed  the 
midterm  to  find  it  out. 

Stating  Objectives  in  Behaviora 1  Terms 

There  is  growing  recognition  of  the  importance  of  stating 
course  objectives  in  behavioral  terms.     If  you  plan  to  teach 
in  Florida,  you  will  almost  certainly  become  better  acquainted 
with  this  facet  of  the  behavioral  approach  to  instruction. 
Florida  school  teachers  are  now  engaged  in  revising  public 
school  curricula  along  these  lines.     Initially,  there  was  a 
mandate  to  complete  the  job  by  December,  1969.    Now  the  time 
period  has  been  extended  to  the  end  of  the  current  school  term. 
The  point  is  that  we  are  talking  about  something  that  is  very 
much  here  and  now.     Your  exposure  to  a  behavior  analytic  teach- 
ing system  will  put  you  in  a  good  position  to  judge  its  worth, 
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to  understand  its  limitations  and  to  work  out  its  logical  ex- 
tensions when  ycu  have  a  class  of  your  own, 

Verbal  Responding  the  Operational  Unit 

Ideally,  the  teacher  should  be  able  to  say  exactly  what 
the  student  will  be  able  to  "do"  after  completing  the  course. 
In  this  course  the  expectation  is  that  you  will  be  able  to  talk 
about  exceptional  children  with  the  facility  of  people  who  reg- 
ularly work  in  the  field  of  Special  Education.    The  tricky  part 
is  to  translate  this  kind  of  statement  into  operational  terms, 
We  have  chosen  verbal  responding  as  the  most  suitable  operation- 
al unit. 

Obviously,  responding  to  stimulus  items  isn't  the  same  as 
spontaneous  shop  talk  in  the  natural  environment.     But  it  comes 
closer  than,  say,  a  ^0  item  multiple  choice  test,  which  is  what 
we  used  last  year  at  this  time.    And  it  has  this  singular  ad- 
vantage:    It  is  something  we  can  do  in  the  classroom  setting. 
Evaluating  the  facility  of  your  on-the-job  performance  in  the 
real  world  is  what  we  hope  to  do  ultimately,  but  we  simply 
haven't  reached  that  degree  of  sophistication  yet. 

Hence,  we  will  infer  your  ability  from  performance  samples 
taken  continuously  over  the  next  ten  weeks.     Your  rate  of  cor- 
rect responding  will  tell  something  about  your  grasp  of  the 
material  (the  better  you  know  it,  the  faster  you  go);  and  the 
incorrect  rate  will  tell  something  about  the  accuracy  and  com- 
pleteness of  it.     Taken  together,  they  should  render  a  fairly 
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good  account  of  what  you  are  learning  from  week  to  week. 

Research  Function  of  the  Precision  Class 

Surely  you  recognize  that  we  wouldn't  go  to  all  this 
trouble  just  to  figure  out  what  grade  to  give  you.  College 
instructors  have  been  giving  grades  since  the  beginning  of 
time.'  The  science  of  grade  giving  is  not  the  issue  here — 
the  application  of  learning  principles  to  the  classroom  is . 
We  are  actively  engaged  in  research  that  may  help  to  improve 
the  quality  of  classroom  teaching.    Consequently,  what  you  do 
this  quarter  may  be  of  greater  than  usual  importance.  With 
your  cooperation  and  a  little  bit  of  luck,  we  stand  a  chance 
of  making  a  contribution  to  the  science  of  teaching. 

Your  participation  in  the  project  requires  nothing  ex- 
traordinary.   The  time  requirement  for  this  course  is  no 
greater  than  for  others.    In  fact,  it  may  be  less,  for  there 
are  no  papers  to  be  written,  no  abstracts,  no  outside  reading 
assignments,  no  cramming  for  midterm  or  final  exams.     If  any 
special  demand  is  made,  it  is  that  you  try  to  meet  your  per- 
formance sessions  on  schedule.     There  are  several  reasons  for 
this,  but  none  more  important  than  the  involvement  of  others 
— your  manager,  the  student  you  manage,  or  both. 

The  only  other  thing  we  would  ask  is  that  you  try  to  ap- 
proach this  course  in  the  spirit  of  scientific  inquiry.  The 
cornerstone  of  the  entire  project  is  the  collection  of  accu- 
rate data.     One  of  the  questions  we  will  try  to  answer  is 
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whether  a  random  assortment  of  inexperienced,  subjectively 
involved  students  can  carry  out  data  collection  as  satisfact- 
orily as  experienced,  relatively  objective  students  can.  The 
conclusions  we  reach  on  this  question  and  others  can  be  no 
more  valid  than  the  data  on  which  they  are  based. 

We  can  only  ask  that  you  take  this  seriously  and  do  as 
conscientious  a  job  as  possible.    The  rest  is  up  to  you. 
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Rules  of  the  Game 

1.  The  student's  grade  will  be  based  entirely  on  20  perform- 
ance sessions.    All  sessions  are  of  equal  weight, 

2.  Performances  will  be  of  the  verbal  responding  type,  except 
for  PS's  10  and  20.     These  will  be  written  and  an  adjustment 
will  be  made  to  compensate  for  the  longer  time  required  for 
writing, 

3.  A  standard  performance  time  of  five  minutes  will  be  used 
for  the  18  verbal  performances.    The  two  written  performances 
will  be  of  30  minutes'  duration, 

4.  Sixteen  performance  units  will  cover  new  material,  in  ac- 
cordance with  the  class  schedule.    The  two  written  PS's  will 
be  review.    Performance  10  will  be  composed  of  a  random  sel- 
ection of  items  taken  from  the  first  eight  units  (Book  1-5, 
Lecture  1-3);  performance  20  will  be  made  up  the  same  way  to 
cover  the  last  eight  units  (Book  6  -  10,  Lecture  k  -  6).  The 
open  units  for  PS's  9  and  15  are  for  repeating  the  performance 
session  on  which  the  lowest  ratio  of  correct  to  incorrect  was 
made.    PS  15  may  not  repeat  the  same  unit  as  PS  9, 

5.  Following  PS  10,  absolute  criteria  for  the  grade  of  A  will 
be  determined.  Feedback  for  the  first  half  of  the  course  will 
consist  of  group  data  on  each  performance  session, 

6.  Performance  sessions  must  be  kept  as  uniform  as  possible. 
Each  item  must  be  read  in  entirety,  with  the  word  "blank"  read 
in  where  blanks  appear.     If  the  manager  cannot  understand  what 
is  being  read  because  of  slurring,  excessive  haste,  or  silent 
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.reading,  he  should  require  the  student  to  read  the  card  again. 

7.  After  the  response  has  been  given,  the  students  flips  the 
card,  reads  the  correct  ansv/er  aloud  and  places  the  card  in 
the  correct  pile  if  his  answer  was  clearly  correct;  or  in  the 
incorrect  pile  if  his  answer  was  clearly  wrong.    If  there  is 
any  doubt  as  to  which  pile  the  card  should  go  in,  it  is  held 
aside  by  the  manager  (proctor)  until  after  the  PS  is  complete. 
No  discussion  of  answers  should  take  place  while  time  is  run- 
ning. 

8.  The  manager  will  review  all  incorrect  and  doubtful  items 
following  the  performance.    The  student  may  take  notes  at  this 
time  but  copying  of  the  items  is  not  allowed.     The  proctor's 
skill  in  providing  clarification  and  helpful  discussion  comes 
into  play  at  this  point, 

9.  It  is  the  proctor's  responsibility  to  see  that  each  per- 
formance session  is  conducted  according  to  the  rules.    His  cer- 
tification to  this  effect  validates  the  data  entered  for  each 
performance  session, 

10.  It  is  the  responsibility  of  the  student  who  acts  as  man- 
ager first  to  make  sure  that  he  takes  the  alternate  form  of  the 
unit  for  his  own  performance;  i.e.,  if  blue  cards  are  used  for 
his  protege",  then  he  must  use  green  cards  for  his  own  perform- 
ance . 
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Date    PS  #      PS  Unit  Lecture  Topic  Assignment* 


Jan  5 
6 

7 

8 

1 

Book  1 

Course  Organization 
Course  Overview 

Ch     1 •   nn  ^-"H 
«««   - >   pp»   J  JJ 

12 

13 
Ik 

1 5 
*  > 

2 
j 

Lecture  1 
Book  2 

Sensory  Handicaps 

H  II 

Ch.  6,  7? 

19 
20 

22 

4 

Lecture  2 
Book  T 

Intellectual 

Deviations 

Ch.  8,  9? 

pp.  lyo-cs-o 

26 
27 
28 
29 

✓ 

6 
7 

Book  4 
Le  c  tura  3 

Intellectual 

Deviations 

Ch.  3f  PP,  35-102 

Feb  2 

3 
4 
< 

8 
9 

10 

Book  5 

Open 
Exam  I 

i*ij.u  icrin  neviSi/ 

Ch.  4,  5; 

pp.  105-148 

9 
10 
11 
12 

11 

Book  6 

Review  &  Discussion 

Learning 

uisaDu.i  uies 

Ch.  2:  pp.  35-81 

16 

1? 
18 
19 

12 
13 

Book  7 

Xf5P"t*137*P  & 

Learning 

Disabilities 

Ch.  10; 

pp.  242-274 

23 
24 
25 
26 

14 

1 5 
* 

Book  8 

Behavior  Disorders 

Ch.  12; 

pp.  294-327 

Mar  2 
3 

16 

Book  9 

Behavior  Disorders 
H  ii 

Ch.  13; 

PP.  330-362 

j 

17 

9 
10 

11 

12 

18 

19 

Book  10 
Lecture  6 

Summary  &  Prospects 
•I  ii 

Ch.  11,  14; 

PD.  276-292, 

365-386 

17 

20 

Exam  II 

Introduction  to  Exceptional  Child  Education 
V.' inter  Quarter e  1970 

*  All  assignments  in  Kirk,  Samuel  A.     Educating  Exceptional 
PAildrpn,     Boston:  Houghton  Mifflin  Company,  1962, 
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Computation  and  Bookkeeping  Procedures 

(a)  Individual  Performance  Record  -  Raw  Data 

(b)  Performance  Adjustment  Sheet 

(c)  Adjusted  Performance  Record 
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COMPUTATION  AND  BOOKKEEPING  PROCEDURES 

The  system  was  designed  to  provide  multiple  computation 
checks  at  each  step.    This  was  considered  a  basic  necessity 
in  a  project  that  required  so  many  different  individuals  to 
make  computations.     The  design  and  function  of  each  of  the 
three  data  sheets  are  explained  below. 

Individual  Performance,  Record  -  SSSLJBeIS 

The  completion  of  the  Raw  Data  sheet  was  the  responsibil- 
ity of  the  student  proctors,    However,  there  were  several  ref- 
erence  points  through  which  the  instructor  might  check  the 
accuracy  of  recorded  data,    The  item  cards  had  been  well  shuf- 
fled prior  to  numbering  in  order  to  eliminate  any  sequence 
cues,  but  after  they  were  in  the  hands  of  the  students,  they 
remained  in  numerical  order.    The  proctor  would  cut  the  deck 
of  cards  prior  to  the  performance  session  so  that  each  student 
would  start  at  a  different  place  in  the  deck.     Then  he  would 
enter  the  number  of  the  top  card  in  the  column  titled,  Frst 
Card  Nr.    At  termination  of  the  five  minute  timed  sample,  the 
number  of  the  last  item  completed  would  be  entered  in  the  col- 
umn titled  Last  Card  Nr.     This  provided  a  check  against  the 
total  number  of  items  counted  by  the  proctor. 

As  noted  in  Chapter  2,  the  use  of  a  standard  five  minute 
performance  time  simplified  rate  computation  by  making  it 
possible  to  double  the  number  of  items  and  move  the  decimal 
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one  place  to  the  left.    After  computing  raxes  on  the  raw  data 
sheet,  students  were  able  to  confirm  the  accuracy  of  their 
calculations  by  summing  the  rates  correct  and  incorrect.  If 
the  sum  did  not  match  the  card  rate,  an  error  in  computation 
was  indicated. 

The  column  titled  Time  in  Mins  usually. carried  a  stan- 
dard 5.0  entry.  Some  of  the  faster  students,  however,  finish- 
ed the  performance  units  in  less  than  the  standard  time.  This 
made  no  difference  in  the  validity  of  the  figure  for  rate,  ex- 
cept to  make  its  computation  slightly  more  difficult  (dividing 
by  ^.5  or  4.33 »  for  example,  instead  of  simply  doubling  the 
figures  entered  in  the  Totl  Itms.  Nr  Corr.  and  Nr  Incr  columns). 

Incorrect  item  numbers  were  sometimes  helpful  in  resolv- 
ing the  occasional  computation  errors  that  occurred  in  the 
early  stages  of  the  project.    For  example,  it  was  possible  to 
locate  some  discrepancies  by  comparing  the  entry  in  the  Nr  Incr 
column  with  the  items  actually  listed  in  the  Incorrect  Item 
NsaM£§  column.    Again,  it  occasionally  happened  that  the  same 
number  would  be  entered  twice  in  the  Incorrect  Item  Numbers 
column,  resulting  in  an  erroneous  incorrect  rate  and  imbalance 
between  the  Card  Rate  entry  and  the  sum  of  Hate  Corr  plus  Rate 
Incr  entries,, 

The  instructor  was  able  to  accept  the  entries  made  by 
student  proctors  with  increasing  confidence  after  the  first 
few  performance  sessions.     In  the  early  stages,  many  errors 
(20  to  30)  were  detected.     Of  these,  only  two  could  not  be  re- 
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solved  through  inspection  of  the  raw  data  sheet.    Both  of 
these  occurred  because  the  cards  were  not  restored  to  numer- 
ical order  after  being  used  by  the  First  Perform  group.  As 
a  result  of  this,  two  students  in  the  Second  Perform  group 
showed  raw  data  entries  that  would  not  balance  on  the  First 
Card-Last  Card  test.    However,  the  figures  for  total  items 
equaled  the  number  of  correct  plus  incorrect,  and  the  stu- 
dent proctors'  counts  were  accepted  as  accurate. 

Performance  Adjustment  Sheet 

Adjustment  of  raw  scores  was  simplified  by  the  use  of  a 
special  purpose  program  written  for  the  Olivetti  Programma 
desk  top  computer.    The  Performance  Adjustment  Sheet  was  used 
in  the  following  manner. 

Step  1  -  The  unit  number  and  adjustment  factors  were  en- 
tered in  the  spaces  provided  at  the  top  of  the  sheet.  Raw 
data  for  each  student  were  entered  in  columns  (a),  (b)  and 
(c)  in  the  manner  shown  in  Figure  6  (p.  ko) .    The  adjustment 
factor  entered  in  the  column  entitled  Adj  Fctr  was  determined 
by  reference  to  whether  the  card  numbers  on  the  raw  data  sheet 
were  even  or  odd. 

Step  2  -  V/hen  the  data  for  all  30  students  was  entered 
in  the  first  four  columns  of  the  sheet,  the  figures  were  fed 
into  the  desk  computer  by  rows  and  groups.    Weighted  means 
were  provided  for  each  group.    The  column  sums  upon  which 
they  were  based  were  entered  in  the  spaces  to  the  right. 


Unit 


PERFORMANCE  ADJ  LISTEN!  SHEET 
Adjustment  Factors:     Green  (odd). 
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  Blue  (even). 
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These  could  be  combined  to  produce  weighted  means  between 
Classroom,  and  Control  groups  and  for  the  entire  class, 

Step  3  -  The  output  for  each  student  consisted  of  the 
products  of  the  adjustment  factor  and  each  raw  datum.  These 
were  entered  in  the  columns  titled  Ad.i  Card  Rate,  Ad.i  Rate 
Corr.  and  Ad.i  Hate  Incr.    An  easy  accuracy  check  was  again 
available  by  summing  the  weighted  means  (Card  Rate  =  Rate 
Correct  +  Rate  Incorrect).     If  the  means  for  the  total  class 
(bottom  right  corner)  were  in  agreement,  it  could  be  assumed 
that  all  the  data,  from  collection  of  raw  performance  figures 
through  adjustment  and  summation  by  groups,  were  accurate.  If 
there  was  a  discrepancy,  the  error  could  be  traced  by  refer- 
ring first  to  groups  and  then  individuals  within  groups.  At 
each  level,  group  or  individual,  the  procedure  was  the  same. 
It  was  necessary  only  to  find  the  inequality  between  card  rate 
and  the  sum  of  rates  correct  plus  incorrect. 

Adjusted  Performance  Record 

Reference  to  Figure  6  shows  the  procedure  by  v/hich  the 
adjusted  data  from  Performance  Adjustment  Sheets  were  trans- 
ferred to  the  Adjusted  Performance  Record,    The  Adjusted  Per- 
formance Record  was  kept  in  the  individual  student's  folder 
so  that  he  could  inspect  the  data  as  well  as  the  performance 
graph  during  performance  periods.     The  three  adjusted  figures 
were  placed  in  the  matching  columns  on  the  adjusted  perform- 
ance sheet.    A  fourth  figure,  Adjusted  Dif ferenceJjatR ,  was 
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obtained  by  subtracting  the  incorrect  from  the  correct  figure. 
Hence,  the  starting  point  in  each  row  of  data  would  be  four 
adjusted  figures  for  card  rate,  correct  rate,  incorrect  rate, 
and  difference  rate.    The  center  column  of  each  of  the  three - 
column  blocks  was  used  for  cumulating  the  figures  in  the  first 
columns  the  third  column,  for  recording  the  mean  cumulative 
performance  (center  column  divided  by  the  number  of  perform- 
ance sessions). 

As  in  the  foregoing  examples,  the  accuracy  of  computations 
could  be  checked  by  summing  means.     In  this  case,  the  sum  of 
the  mean  difference  rate  and  the  mean  incorrect  rate  should 
equal  the  mean  rate  correct,  and  this  figure  plus  mean  incor- 
rect rate  should  equal  the  mean  card  rate, 

Figures  for  group  performance  were  available  also.  The 
weighted  means  provided  on  the  Performance  Adjustment  Sheet 
could  be  accumulated  in  the  same  fashion  as  for  individuals. 
The  instructor  was  able  to  follow  the  progress  of  groups  as 
well  as  individuals.    In  the  present  experiment,  group  moni- 
toring was  not  required.    However,  the  potential  for  following 
the  effects  of  experimental  manipulations  on  continuously  re- 
corded dependent  variables  should  be  clear. 


APPENDIX  C 


The  Relationship  .Between 
Item  length  and  Performance  Rates 

Table  A-l 
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TK5  RELATIONSHIP  BETWEEN  ITEM  LENGTH  AND  PERFORMANCE  RATES 

Table  A-l  shows  three  columns  of  30  correlation  co- 
efficients each.    The  coefficients  represent  the  relation- 
ship between  mean  lines  per  blank  (LP3)  of  the  particular 
performance  units  that  each  individual  took,  and  the  per- 
formance rates  achieved  on  those  units.     The  relationship 
betv/een  card  rate  and  LP3  is  strong,  with  17  of  30  students 
showing  a  reliable  negative  correlation  between  item  length 
and  number  of  cards  attempted.    All  but  one  of  the  coeffici- 
ents is  preceded  by  a  negative  sign.    Correct  rate  shows 
only  six  students  with  reliable  coefficients.    However,  the 
influence  of  item  length  can  still  be  seen  in  the  fact  that 
all  but  four  of  the  coefficients  are  negative.    The  relation- 
ship becomes  most  attenuated  v/ith  difference  rates.    It  will 
be  recalled  that  the  difference  rate  includes  the  effect  of 
incorrect  rate  as  well  as  correct  response  rate.     It  appears 
that  the  relationship  becomes  weaker  as  additional  sources 
of  variance  impinge  upon  it.     This  is  as  expected. 

The  relationship  between  item  length  and  performance 
measures  persists  despite  the  reduction  of  variance  in  the 
items  that  were  used  during  the  pilot  project.  The  level 
of  the  relationship  is  noticeably  lower,  however.  Few  of 
the  coefficients  in  the  present  series  reach  the  level 
based  on  mean  group  performance  in  the  pilot  project.  In 
short,  the  technique  worked  as  intended  and  the  unv/anted 
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relationship  can  be  further  reduced,  presumably,  by  striv- 
ing for  still  greater  uniformity  of  items  between  units. 

The  use  for  which  the  items  are  intended  should  govern 
the  amount  of  attention  that  is  paid  to  item  length.  For 
general  classroom  purposes,  an  awareness  of  the  relation- 
ship.between  item  length  and  performance  rates  is  probably 
all  that  is  required  for  satisfactory  results.     The  use  of 
unadjusted  difference  rates  in  the  present  study,  for  ex- 
ample, probably  would  have  worked  no  hardships  on  the  stu- 
dents, and  certainly  would  have  spared  the  instructor  the 
burdensome  task  of  performing  the  adjustment  computations. 
For  research  purposes,  however,  the  experimenter  wants  to 
control  as  many  sources  of  variance  as  possible,    When  rate 
is  used  as  the  basic  datum — i.e.,  when  the  tests  are  timed 
—the  factor  of  item  length  can  be  important. 
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TABLE  A-l 

Rank  order  of  Pearsonian  correlation  coefficients  by 
unadjusted  performance  rates  of  individual  students  and 
mean  lines  per  blank  of  items  included  in  their 
respective  performance  rates 
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APPENDIX  D 

Raw  Data  Used  in  Data  Analyses 
in  This  Study 

Key  to  Data  Tables 
Data  Tables 
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K5Y  TO  DATA  TABLES 
The  tables  are  identified  by  student  number.  Refer- 
ence to  Figure  1  (p.  22)  shows  the  Classroom  group  divided 
into  subgroups  (A)  and  (B)  for  purposes  of  controlling 
practice  benefit.    Subgroup  (A)  was  comprised  of  students 
01  through  07;  subgroup  (B),  students  03  through  lk.  The 
Control  group  was  composed  of  students  15  through  30. 
Column  codes  are  as  follows: 
Card  =  Card  Rate 
Corr  =  Rate  Correct 
Incr  =  Rate  Incorrect 
Fctr  =  Adjustment  Fa.ctor 
Rov/s  represent  performance  units.    They  are  not  label- 
ed but  conform  to  the  standard  format  shov/n  on  the  following 
page . 


Standard  format  for  rgWg 


Book  Unit  1 
Lecture  Unit  1 
Book  Unit  2 
Lecture  Unit  2 
Book  Unit  3 
Book  Unit  4 
Lecture  Unit  3 
Book  Unit  5 
Book  Unit  6 
Book  Unit  7 
Lecture  Unit  ^ 
Book  Unit  8 
Book  Unit  9 
Lecture  Unit  5 
Book  Unit  10 
Lecture  Unit  6 

Repeat  Unit  1 
Repeat  Unit  2 

Written  .3xam  1 
Written  ibcan  2 
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